That's a really smart question! When you're dealing with automation components where downtime can cost millions, price becomes just one piece of the puzzle. Here are the key metrics I'd track beyond cost:

1. Quality Metrics - This is crucial when failures are expensive. Track defect rates (percentage of defective parts), RMA rates (returned materials), and how often suppliers meet your quality specifications. A single bad component can shut down your entire production line.

2. Delivery Reliability - Since lead times are unpredictable, focus on on-time delivery rates and fill rates (how much of your order they can actually deliver). Also track lead time variability - how consistent their delivery windows are, not just how fast they promise.

3. Technical Support & Responsiveness - How quickly do they respond to technical issues? What's their average resolution time? When you're facing downtime, hours matter, and good technical support can save you millions.

4. Supply Chain Resilience - Look at their inventory levels, backup suppliers, and contingency plans. Suppliers with robust supply chains are less likely to leave you hanging during disruptions.

5. Documentation & Traceability - Good suppliers provide detailed documentation, certifications, and component traceability. This helps with troubleshooting and ensures you're getting exactly what you need.

6. Financial Health - A financially stable supplier is less likely to disappear or cut corners. Check their credit ratings and financial stability.

The bottom line? When downtime costs millions, you want suppliers who are reliable partners, not just low-cost vendors. A slightly higher price from a reliable supplier can save you millions in avoided downtime!