Manufacturing Update
Hey everyone, Ryan here.
Since our last update we’ve been focused on manufacturing Arsenal 2s and I wanted to provide an update on the progress there (and a few other things). So let’s jump in.
Manufacturing Update
The bad news first: We hit a production snag. We’re not here to make excuses, but if you’re curious as to the details, keep reading. If not, the updated timeline is below in the Schedule section.
The details: Mass Production for Arsenal 2 began almost three weeks ago. Unfortunately, a few days into Mass Production we started receiving reports from our contract manufacturer that the failure rate on our Flash/QA rigs was abnormally high and growing.
You might remember from previous updates that the Flash/QA rig is a custom test suite that automatically boots the Arsenal 2 over USB, performs a series of high level software tests to confirm the hardware performs as it should, and then writes the initial firmware image to memory.
Normally we are on site for the beginning of mass production, but due to COVID restrictions overseas we were not able to be there for Arsenal 2. Unfortunately, debugging over video chat and email proved to be ineffective, so we decided to temporarily halt production and have our contract manufacturer ship a number of failing units to us for review (with customs and COVID delays, it takes a week to get them here even when we pay for the fastest shipping). These units proved to be challenging to debug, but in the end we found two issues:
1) eMMC Linux driver Implementation:
The Arsenal 2 hardware contains an eMMC chip which provides non-volatile memory for the system. The eMMC chip stores the system image and is also used to store data while Arsenal 2 is running some of its photo processing algorithms. Ensuring this chip runs at the highest speed possible is critical to ensure quick boot times and fast image processing. This means the eMMC bus speed needs to be configured to run at 200MHz, which is the maximum allowed frequency (specifically it’s called HS400).
Several units were not properly configuring at HS400 and failing on the Flash/QA rig. All of the conventional hardware debug tricks proved to be ineffective in finding the root cause of this issue. We ultimately had to start adding debug statements to the Linux driver. Once we did, we found that the initialization failed every time the processor attempted to change the bus frequency from 50MHz to 200MHz using the SWITCH Command (CMD6 for all you hardware junkies). After digging deeper and reverse engineering the driver using the Embedded Multi-Media Card Electrical Standard (350 exciting pages defining how eMMC operates and performs), we found a discrepancy between the Standard and how the driver switches to the HS400 mode. The specific problem was the “Tuning Process” was being conducted during the wrong configuration step. Some minor adjustments to the order of operations in the driver allowed these units to start working as expected.
If we hadn’t gotten this to work properly, we would have had to run at High Speed SDR. From the table, you can see there’s an ~x8 performance boost by getting the eMMC to properly function in HS400.
2) The power of 20mV:
All manufacturing processes have some tolerances that are considered acceptable. The manufacturing of integrated circuits (ICs) is no different. Due to the global chip shortage, we received ICs with date of manufacture codes ranging from 2017 to 2021 (normally you just get a bundle of everything from the same date, but COVID). Most of the development and testing was done on chips from 2017 (for no particular reason besides parts from this year were just easier to obtain early in the development process). Most of the ICs used in Mass Production are from 2020 and 2021. What we’ve found is some of these ICs require one of the voltage rails to be slightly higher than ICs from previous years. “Bumping” the voltage rail slightly allowed these units to boot and test without issues. The increase is extremely small, 20mV to be exact, which is just twenty thousandths of a volt. This is still well within the recommended operating conditions listed in the IC’s datasheet. Luckily the DCDC converters on Arsenal 2 are software programmable, so no parts had to be adjusted or replaced.
We were a bit surprised to see these two issues happen, given that we didn’t see either at all during our Pilot Run. Turns out, the slight variation in sourced components (based on having the same components but with different manufacture dates) was the issue.
The good news here is that all of the hardware components are testing fine themselves. We simply had to find and make some changes into how the board is programmed to fix these issues. A true hardware issue would have caused a much longer delay.
Schedule Update
We’ve updated the Flash/QA rigs and we’re happy to report that Mass Production is back up and running smoothly the last few days.
The new manufacturing schedule ends up being a three week delay. On top of that, we had trouble securing shipping logistics for the new dates (likely due to the holidays) and that added another week to the schedule.
The short version: based on all of the above, our new shipping target is to begin shipping directly to backers on December 15. With the current manufacturing and logistics schedule, we expect to have the last Arsenal 2 backer rewards shipped by January 14th.
Before moving on, I do want to say thanks to Phillip and Jack for their tireless efforts to find the root cause and solve the manufacturing issues. While nobody is happy with the impact to shipping, the team did well to solve this as quickly as possible and minimize impact.
Also, I wanted to mention that the feedback on the last few updates was really positive and, while we’re disappointed to see a delay at this point in the game, we appreciate the continued support shown by our backers.
Pledge Manager
All of the pledge manager emails have been sent out to the email address on file with Kickstarter. If you didn’t receive one:
1) Check your email (the email comes from help@witharsenal.com)
2) Check your spam folder (we haven’t had a ton of reports of them going there, but surely a few have)
3) Email help@witharsenal.com and they’ll provide you a link
There are lots of interesting insights when we start to get detailed reward data. A few that popped out to us:
A) The Top 5 most popular cameras from backers so far:
Sony Alpha-A7 III
Canon 5D Mark IV
Nikon D850
Canon EOS R5
Sony Alpha-A7r IV
B) Canon is the most requested brand by a fair amount, followed by Nikon, Sony, and Fuji
C) The phone mount is the most popular accessory so far, with about 75% of backers adding it
Also, if you need to change anything with your pledge manager (given the shipping timeline or for anything else) drop an email to help@witharsenal.com.
New Cameras
The Sony A7 IV announcement was pretty big news recently and many of you asked about it. I had a brief chance to try it out and my first impression was that it’s a great camera and will probably pair really well with Arsenal 2. I have the A7 IV on backorder currently, and once I get it in we’ll evaluate if we can add it for Arsenal 2 (we probably can, but need to test to be sure).
Beyond that, we’ll be evaluating more cameras for support shortly after launch. What cameras would you like to see us add? Feel free to comment below and/or log your votes here. How much demand there is for a camera is one of the inputs when we decide to support a camera, so your feedback definitely helps!
Wrap Up
At this point I’m not expecting any other issues before launch, but should we encounter any we’ll update you ASAP. We’ll also post another update after the November holidays just to let you know we’re on track and manufacturing/logistics are going as planned. In the meantime, we’ll be posting more Arsenal 2 tutorials soon, which will help get you prepared for launch.
Thanks again for all of your incredible support.
- Ryan