Assembly lines for electronics products are complicated beasts, often composed of many custom tools and fixtures. Typically a microcontroller must be programmed with firmware, and the circuit board tested before assembly into the enclosure, followed by functional testing afterwards before putting it in a box. These test platforms can be very expensive, easily into the tens of thousands of dollars. Instead, this project uses a set of 12 Raspberry Pi Zero Ws in parallel to program, test, and configure up to 12 units at once before moving on to the next stage in assembly.
Fixing Fixture Bottlenecks
The company where I work, Propeller Health, develops IoT products that are assembled in a way similar to many other companies; there is a circuit board and a plastic enclosure. The bare PCBs go through SMT twice (components on the front and back), then they go through ICT (In-Circuit Test) where they are programmed and pogo pins on each of the test points verify all components on the circuit. They are also given a unique MAC address for Bluetooth based on a 2D barcode sticker placed on the board after SMT. After that they are assembled into the plastics and run through a functional test fixture before they are put into inventory mode and boxed. We have a product line of around a dozen different devices, and each uses a different PCB and enclosure.
Production runs need test and programming fixtures, a topic i wrote about a few years ago. To increase scale and reduce costs we started working more closely with our contract manufacturer to identify and fix the bottlenecks in the assembly line process, and we noticed that the ICT stage was taking 6 minutes for a panel of 6 boards. The ICT section itself was relatively fast, but the firmware programming and mac programming were taking a long time because the test platform was not capable of multiple simultaneous serial port connections.
This was a $30k fixture with 6 Segger programmers, 6 Keysight scanners, sitting on top of a $100k Teradyne platform, and it was our biggest bottleneck and greatest expense. Further, each device in our product line requires different fixturing. The combination of these expensive fixtures and slow cycle time was making our COGS (Cost Of Goods Sold) untenable.
On an assembly line, time is literally money, as every cycle time for every operator on the line is measured and added to the cost of the product. Reducing a cycle by even a second adds up quickly over a volume of thousands. Over 10k units, 2 second reduction is 6 hours of labor, at $15/operator hour is $.01/unit, and CMs, especially US based ones, are charging more than $15/hour. Having a 6 minute cycle time for 6 pieces was ripe for improvement.
Stage One – Arduino
The first stage was more a proof of concept. Could we pull some of the functionality off of the expensive ICT fixture onto a cheaper faster fixture so that we could scale in serial instead of buying another fixture to scale in parallel?
For this stage we implemented a feature in our firmware we call self-test, which is similar to a JTAG boundary scan. Rather than placing pogo pins on every net, we put a feature into the firmware to run a self-test on as many components as it could, and report the results over serial. This way we’d be able to connect only with the power and UART pins and get a complete and very fast readout of what exactly failed. Eight Arduinos connected via a USB hub to a tablet PC with a custom Python script and interface was the solution.
A single 40pin IDE connector was enough to have power, an RGB LED, RX, TX, and a single button that would kick off the cycle for all 8 at once. The Arduino would request a self-test from the DUT (Device Under Test), which would test its components and report back. Tests included verifying that sensors were reading values within an expected range, but also turning on the LED and buzzer and measuring the voltage drop to verify that outputs worked as well.
This worked, and proved the concept of creating a box with most of the electronics paired with a simple customized fixture for the panel, but it wasn’t enough. Pulling out some of the circuit tests was insufficient to reduce time significantly. We needed to pull out more tasks.
Stage Two – Raspberry Pi
One of the more expensive parts of the ICT fixture was the Segger J-Link programmers. If we could pull out programming from ICT with something cheaper, we could reduce fixture costs significantly. Many chip manufacturers offer programming at the factory so that your microcontrollers come on a reel with your firmware pre-programmed. This is useful when you don’t trust your manufacturer or don’t want to have a special fixture for programming, but it does have an added cost, and there’s a lead time, and changing your firmware is more difficult, and you can’t use that part on any other product.
Programming in house had a lot of appeal for us, simplified our supply chain, and gave us more flexibility to change our firmware and shift inventory as needed. Enter the Raspberry Pi Zero W and OpenOCD. The Raspberry Pi is more than capable of running OpenOCD to program a microcontroller, and was surprisingly easy to configure and get running. It also had a serial port and GPIO for controlling WS2812B LEDs, and was small enough that it was easy to put a bunch in a box.
The connector expanded to a second 40pin IDE cable, and support for up to 12 parallel units, the size of our largest panels. Each Pi runs completely independently, without communicating with each other. The only thing they have in common (besides power supply) is that all 12 are connected to the same pushbutton on the front of the fixture. The button is used by the operator to start the programming cycle, but each Pi runs its process independently.
Drawing on the Arduino experience further, the Raspberry Pi process would first program the unit, then verify it was programmed correctly by communicating over the UART, then execute the self-test on the device, which would test the accelerometer, microphone, IR sensors, LED, buzzer, etc, and report back the results and a PASS/FAIL status. We had accomplished our primary goal of removing a significant portion of the process time and cost from the ICT fixture, but we could go further.
Stage Three – Adding Features
With the addition of cameras to the fixture, we could read the MAC address on the sticker and use the serial port to program the MAC address into the unit. This allowed us to tie our process to the MAC address that was used to track the unit through the rest of the assembly line.
If you’re wondering why we didn’t pre-assign numbers and then have the operators put on the correct number, it comes down to reducing the chances for operator error. Giving the operator a sheet of stickers and telling them to put them on the units is way less error-prone than having them put specific stickers on specific units, and faster than printing out barcodes one at a time for individual units.
Next we added the unique key stored on the unit for authenticated communication. The pi generates a 64 byte key, programs and verifies that key on the sensor, then encrypts the key and stores it in the log file. That key gets decrypted on our servers and allows us to communicate with our sensors.
In addition, we added thorough logging; a system log tracks the version numbers of the script, config file, and firmware file, and a separate file logs each cycle with diagnostic data about each stage of the process. These logs are stored on a thumb drive, but also uploaded regularly to a secure FTP so that we can import the logs (and the encrypted keys) immediately, as well as keep tabs on our CM and monitor when they are running our jobs and help them diagnose any problems.
We also made all of this run off a thumb drive. On boot the pi starts a Python script whose sole job is to wait for a thumb drive to be inserted, then run the script on that thumb drive. The thumb drive contains the main Python script, config file, firmware hex file, and logs. This way we have a single box that is completely agnostic to the assembly line, and then we have a dozen thumb drives that stay with the line for that particular device. We don’t need an interface to the pi because we can just plug the thumb drive into a computer to configure it however needed. Anything that makes the job easier for operators makes the assembly line run smoother, and saying “plug these into the back of the box” worked pretty well.
At this point we could program the microcontroller, then execute a self-test, then program the MAC address, and generate a unique key, and output to an LED the status of the process (green for success, yellow for in progress, and red for failure), and save the log to a file. This shaved MINUTES off the cycle time. Before, a panel of 12 was split in two, then each half went through the ICT fixture for 6 minutes, for a total cycle time of around 12 minutes for 12 devices. After the change, we had the full panel of 12 take under a minute, then split and spend under 2 minutes each at ICT. We went from 12 in 12 minutes to 12 in 5 minutes, with more control over the process and greater transparency.
Stage Four – Neptune is Alive
After our first box worked, our CM asked for more so they could service multiple assembly lines, plus have backups and one for test/development. The design was formalized, we found a new enclosure that could be modified easily, and designed a PCB. Total cost for all components is around $500, and the PCBs are easy enough to assemble by hand in a couple hours. We called the project Neptune and the boxes started coming together. Now we have plenty of Neptune boxes, and we have a spec for our fixture developer that allows them to have a very simple circuit they can replicate over and over. Fixtures are rugged beasts that require special knowledge and experience that we don’t have and don’t want to develop, so we outsource that part. The most recent fixture worked immediately with no changes, which is usually nearly impossible.
Stage Five – Changing Cameras
A MAC address sticker is placed on every unit after SMT, but this address must be programmed into the microcontroller itself. To do this, a barcode scanner is used. This is a production environment, and we are packing 12 scanners into a small space, and the scanners must accurately and reliably detect a 2D barcode about 1/2in^2, so not just any scanner will do.
Originally, on the suggestion of our CM, the fixtures were outfitted with Keyence scanners. After we discovered that these are $1400 each, we realized a huge potential for cost reduction. Each of our fixtures has 10-12 devices per panel, so replacing the Keyence scanners with $250 Zebra scanners was an immediate gain. The downside is that our Neptune PCB was designed for RS232 communication with the Keyence scanners using only 2 UART pins, and the Zebra scanners required 5V logic levels with hardware RTS/CTS pins as well. Fortunately we were able to build a cable with an inline circuit that contained an ATtiny84 for each of the cameras.
With a small config file change, we can tell the system to use a slightly different communication protocol. The Pi sends a serial signal, which is stepped up to RS232 inside the Neptune box, then on the cable dongle it steps down to 5V, goes into the ATtiny, where it is parsed, and if the desired signal then it sends a pulse to the trigger pin of the Zebra scanner, which then scans the barcode and sends the result back to the dongle, where it is stepped up to RS232 and heads back to the unit. The dongle meant that we didn’t have to change anything about the box, and the fixture didn’t have any additional circuitry as well.
Results
The results of Neptune have saved the company a lot of money in fixtures and in unit cost, improved our scalability, given us greater transparency in our production, added features, and made development of new features easier. The Raspberry Pi was instrumental in this, as was OpenOCD. We learned a lot and developed some techniques that would likely be valuable to anyone else doing manufacturing of electronic devices. We’re still working out exactly how we can share this without compromising security or trade secrets, but this is our first step.
No comments:
Post a Comment