Prior to issuing the command sequence we tune around the beacon frequency, taking note of the "noise spectrum". No variance in amplitude detected, just clean white noise. At 0310, commenced sending a 12 block sequence, 3 * (Reset,Tx-Off,Reset,Tx-On).

At the completion of the sequence, again tuned around the beacon frequency, and noted a noise peak of 4 to 5 db, in the vicinity of expected beacon frequency. The width of this peak was about 5 kHz. Listened to it for about 15 seconds and then sent a Reset, Tx-Off command and the noise peak disappeared, approximately 2 seconds after the Tx-Off block was completed.

Completed my session with a number of Aux-Batt commands, but did not attempt the Tx again. Range was nominally 12000/15000 kilometres during command period. Was using my last calibrated uplink frequency.

This tends to suggest that the IHU and L Band Rx are operational. Bus voltage insufficient to open that damn relay. Considering SA=-72, the remaining cells are holding up well. Hope this makes your day, like it has mine.

Colin Hurst, VK5HI
for the AO-40 Command Team

"The Parkes Observatory" is operated by the "Australia Telescope National Facility (ATNF)" which is part of "Commonwealth Scientific and Industrial Research Organisation (CSIRO).

From here were received the live pictures of Apollo 11 Moon landing, 1969. See above homepage of Parkes.

The radiotelescope has a diameter of 64 metres and the system noise figure is approx. 25 K.

From about 6:00 UTC, a 4 MHz wide spectrum with center frequency of 1097 MHz was scanned with high level technique in order to find the LO frequency of the AO-40 L1 receiver. A band of approx. +/- 20 kHz around the actual LO frequency was examined intensively. Three hours were needed to prepare the setup for AO-40 and to search.

Unfortunately nothing was heard which could be a signal of AO-40!!

The fact that nothing was heard of the L1 receiver's LO does not lead to the conclusion that AO-40 is completely dead. The receivers of Parkes are not really fitted for frequencies below 1.25 GHz, and naturally one does not know how good the L1 receiver's LO is shielded or how much of the signal would go through the antenna to the outside world. During the next days more research follows.

The L1 command receiver is continuously switched on, and it would have been good news if we had such a confirmation that the 10 V power source for the L1 receivers is still functioning. All receivers and also the IHU are supplied with the 10 V source.

All command stations are continuing to send commands to the satellite in order to switch the batteries.

I would like to say a great thank you on behalf of AMSAT-DL to Brett Dawson, VK2CBD and the team of Parkes observatory, especially to Dr. John Reynolds (Officer in Charge) and John Sarkissian (Operations Scientist).

Peter Guelzow, DB2OS
President AMSAT-DL
 

Attempts are underway to have one or more very large radio telescopes (non- amateur) listen for the receiver local oscillators. These are well shielded and this search may be negative even if the receivers are powered up properly. Results and further details will be reported when they become available.

Tests of the auxiliary batteries at the Amsat lab in Orlando confirm the benign behavior of these cells when subjected to a scenario duplicating that experienced on AO-40. Extra main battery cells are not available for testing. Tests also show that the battery switching relay requires 12 volts to switch reliably. The receivers and IHU require 10 volts for operation. The S2 transmitter was not designed to work below 20 volts.

Some minor changes have been incorporated into the simple machine code commands to switch batteries and receivers in order to optimize their chances of success.

As we have mentioned before, we believe that we have a shorted main battery, and we may have to wait for a cell to open to regain voltage and control. This may be a long-term prospect. Short-term failure does not predict long-term results for this problem. There is no reason to believe that the BCR's are damaged or will be damaged by the shorted battery pack. The BCR's are designed to function with the solar panels extended, when they have to handle over 3 times the power available in spin mode.

The AO-40 Command Team would like to thank all of you for your support and good wishes. When any additional information becomes available we will report it here as quickly as possible.

--W4SM for the AO-40 Command Team ****************************************************************

    AO-40  ATTITUDE - PRECESSION BEHAVIOR / W4SM
       Mystery Effect = -11.800 degs ALON/week
        DATE            ALON   ALAT    SA     ILL%
---------------------------------------------------
  2004  Jan  27 [Tue]  350.0   -3.0   37.8    79.0
  2004  Jan  29 [Thu]  346.3   -3.0   32.5    84.4
  2004  Jan  31 [Sat]  342.6   -2.9   27.1    89.0
  2004  Feb  02 [Mon]  338.9   -2.9   21.8    92.8
  2004  Feb  04 [Wed]  335.2   -2.9   16.5    95.9
  2004  Feb  06 [Fri]  331.5   -2.8   11.1    98.1
  2004  Feb  08 [Sun]  327.8   -2.8    5.8    99.5
  2004  Feb  10 [Tue]  324.1   -2.8    0.5   100.0
  2004  Feb  12 [Thu]  320.4   -2.7   -4.9    99.6
  2004  Feb  14 [Sat]  316.7   -2.7  -10.2    98.4
  2004  Feb  16 [Mon]  313.0   -2.6  -15.6    96.3
  2004  Feb  18 [Wed]  309.3   -2.6  -20.9    93.4
  2004  Feb  20 [Fri]  305.6   -2.5  -26.3    89.7
  2004  Feb  22 [Sun]  301.9   -2.5  -31.6    85.2
  2004  Feb  24 [Tue]  298.2   -2.4  -37.0    79.9
  2004  Feb  26 [Thu]  294.5   -2.3  -42.3    74.0
  2004  Feb  28 [Sat]  290.8   -2.3  -47.6    67.4
  2004  Mar  01 [Mon]  287.1   -2.2  -53.0    60.2
  2004  Mar  03 [Wed]  283.4   -2.2  -58.3    52.6
  2004  Mar  05 [Fri]  279.7   -2.1  -63.6    44.4
  2004  Mar  07 [Sun]  276.0   -2.0  -68.9    36.0
  2004  Mar  09 [Tue]  272.3   -2.0  -74.2    27.2
  2004  Mar  11 [Thu]  268.6   -1.9  -79.4    18.3
  2004  Mar  13 [Sat]  264.9   -1.9  -84.5     9.7
  2004  Mar  15 [Mon]  261.2   -1.8  -87.4     4.5
  2004  Mar  17 [Wed]  257.5   -1.7  -83.7    11.0
  2004  Mar  19 [Fri]  253.8   -1.7  -78.6    19.7
  2004  Mar  21 [Sun]  250.1   -1.6  -73.4    28.6
  2004  Mar  23 [Tue]  246.4   -1.6  -68.1    37.3
  2004  Mar  25 [Thu]  242.7   -1.5  -62.8    45.7
  2004  Mar  27 [Sat]  239.0   -1.5  -57.5    53.7
  2004  Mar  29 [Mon]  235.3   -1.4  -52.2    61.2
  2004  Mar  31 [Wed]  231.6   -1.3  -46.9    68.3
  2004  Apr  02 [Fri]  227.9   -1.3  -41.6    74.7
  2004  Apr  04 [Sun]  224.2   -1.2  -36.3    80.5
  2004  Apr  06 [Tue]  220.5   -1.2  -31.1    85.7
  2004  Apr  08 [Thu]  216.8   -1.2  -25.8    90.1
  2004  Apr  10 [Sat]  213.1   -1.1  -20.5    93.7
  2004  Apr  12 [Mon]  209.4   -1.1  -15.2    96.5
  2004  Apr  14 [Wed]  205.7   -1.0   -9.9    98.5
  2004  Apr  16 [Fri]  202.0   -1.0   -4.6    99.7
  2004  Apr  18 [Sun]  198.3   -1.0    0.6   100.0
  2004  Apr  20 [Tue]  194.6   -0.9    5.9    99.5
  2004  Apr  22 [Thu]  190.9   -0.9   11.1    98.1
  2004  Apr  24 [Sat]  187.2   -0.9   16.4    96.0
  2004  Apr  26 [Mon]  183.5   -0.9   21.6    93.0

With regard to the stability of the attitude/spin, this will not be a concern for a very long time. We are currently rotating at 3.5 RPM. The spin decay rate is extremely slow. It will take approximately 4 years to drop this to 3.0 RPM. We can magnetorque at speeds as low as 1.5 RPM. The mystery effect will decrease ALON approximately 11.5 degrees/week. It does not affect ALAT, though ALAT will change slightly as the orbit precesses.

MAIN BATTERY NOTES (and conjecture): The main batteries consist of three packs housed in sheet aluminum cases and bolted to the radial braces between panels 1/6, 2/3 and 4/5. The cells within the packs have threaded metal binding posts and the cells are connected by thick metal straps with strain relief "U's" in them. The pack at 2/3 consists of 7 cells and is the negative end of the chain. The pack at 1/6 consists of 6 cells and is in the middle of the chain. The pack at 4/5 consists of 7 cells and is at the positive end of the chain. The main battery pack at 1/6 is the closest battery to the "flaky" heat pipe thermistor, though it is located "below" this heat pipe near the omni end of the spacecraft. Main battery packs 4/5 and 1/6 lost their thermistors during the 400N incident. Whether this was due to trauma to the battery or damage to the cabling is unknown. If a short to ground occurred in the 1/6 battery pack it would pull the cells on the negative side of the short in this pack to zero, as well as all cells in the 2/3 pack. Depending on the location of the short and the status of the cells in pack 4/5, this could pull the main buss voltage to half normal (14 volts) or even 10 volts or below. A short at this location might have generated enough localized heat (or even some hot metal spatter) to damage the thermistor on the flaky heatpipe or, more likely, its wiring. This is appealing because it would represent a single point failure, rather than a failure cascade. One piece of evidence that doesn't clearly fit with this theory is that the cells in pack 2/3, the one main battery pack that still has a thermistor temperature sensor, do not appear to get warm following the voltage drop. We do not know how much capacity remained in these cells. It is possible they contained relatively little energy.

As several of you indicated, we are in a waiting game for the main battery to develop one "open" cell.

--W4SM for the AO-40 Command Team

Our hypothesis remains that we have a battery failure clamping the voltage low. We're looking at the spacecraft physical and electrical layout and even the "flaky" heatpipe reading to see if we can come up with a more detailed failure point explanation.

If our theory is correct, the remaining cells of the main battery are taking quite a beating as the BCR's try unsuccessfully to bring them up to the set point voltage of about 28 volts. We expect that more than 6 amps are flowing through the remaining cells in the main battery and this current will increase by about 30% as the solar angles improve. We only need for one of these cells to lose enough electrolyte to "open" and we could be back in business. I'll repeat my comment from before that time is an unknown here and days or weeks without result do not mean ultimate failure.

I will be out of town for two days on an unavoidable business trip, but will update the bulletin board when I return on Sunday. The command team will continue to try to regain the S2 and K Tx's and report any successes here.

-W4SM for the AO-40 Command Team

If the main battery has us clamped at a voltage where the above will not work, we will have to wait to see if solar angle improvements help, or if one of the bad cells eventually goes "open".

Although we have not had time to read the bulletin board much, and we can't claim to fully understand what has happened, it is clear that operator practices, though not always perfect, had absolutely nothing to do with this event. The fact that the passbands cut off on Sunday was an example of the kinds of safety features that we have in place to protect the electronics on board. There are also input AGC's and output limiters. AO-40 was designed to withstand all that you can throw at it. Although there was a lot of passband activity on Sunday, and under really marginal conditions heavy usage could put us in transiently a negative power budget, it is now clear that it was the failing of another cell on the main battery that caused the passband shutdown at that time.

The main batteries consist of 20 40 amp-hour cells arranged on three of the radial support arms inside the spacecraft. There are two packs of 7 cells and one pack of 6 cells. It is entirely possible or even probable that the main batteries suffered some damage during the 400N motor event. One clue to this is the fact that we lost two of the three temperature sensors attached to these packs, and since that time we have been able to measure the temperature of only one of the three packs. It may well be that this was the only "normal" main battery pack.

One mystery in the telemetry is where the "energy" is going from this battery failure. With the exception of the "heat pipe" spike, which we believe is not real as discussed before, there are really no changes in temperatures anywhere over the next 15 minutes. The answer MAY be that the bad cells in fact had deteriorated to the point that they had very little storage capacity left and so had little energy to dissipate. The aux. bat. was probably only a few percent charged at the pre-event voltage and had little energy to contribute to the event. The temperature on either of its two packs does not change during this time, supporting this conclusion.

The above (except the operator practices part), is our best understanding of things at the moment and is always subject to change. We'll keep you posted.

--W4SM for the AO-40 Command Team

PERSONAL NOTE : Like my colleagues on the command team, I have lived and breathed AO-40 for over four years. All of us can almost mentally decode 400 bps PSK. We hear it in our sleep. I was watching the battery voltage telemetry at the exact moment that the voltage dropped precipitously. In my "day job" I have frequently watched catastrophic events unfold in human beings, and the feeling was EXACTLY the same. Part of my day job is to have to make quick decisions during times such times, decisions that can have serious consequences. I was instantly aware that we had a serious power event, and I considered cutting the main battery loose and trying to run on the nearly completely discharged and untested under load aux. battery. However, because I did not have a clear understanding of where the primary fault was, I elected to watch things and try to figure out what was happening. The general rule of, "when in doubt wait to understand" works most of the time... In this case it didn't, and I'd sure like to live those minutes over again and cut the main battery loose. Hindsight is always 20-20. Of course, if it had crashed anyway, then I'd really be beating myself up. If it's at all possible to bring AO-40 back, we will. If the voltage is clamped low and there is no other damage, we may end up waiting a long time for a cell to "open", hopefully not as long as for AO-07. ..or it may happen today. No success for even weeks or months does not mean that we won't eventually be successful. We will sure keep trying. Several of you have written very nice notes of support. Thank you.
 

By way of review, the command team had been watching the main battery for a number of months and noting that its behavior, while not alarming, was not fully as expected. In particular, there were voltage fluctuations with spin under relatively light power loads. When panels 1-6 received sunlight their 1/2 complement of solar cells would often lead to a voltage dip, ; which would pop back up when the fully arrayed panels were in the sunlight. This suggested that perhaps one cell in the 20 cell main battery might be weakening. On Sunday, we noted that the battery was not maintaining its set voltage and this led to the safety software shutting off the passbands. Some minor adjustments were made to the solar array voltage offsets and this helped some, but we stayed in passband off mode to study the matter further. It appears that at this time we may have acquired a second bad or shorted cell, though we are analyzing this further. During eclipse on orbit 1487, at approximately 1930 utc on 2004-01-26 the battery voltage hit the extreme low voltage trigger. This caused safety resets of the battery and solar array set points and the auxiliary battery was tied in to the main battery. This all appears to have happened benignly, and when I got AOS at MA=8, about an hour and a half later on the following orbit, I noted that the extreme low voltage flag had been set ,as well as the other changes noted. Voltages on both batteries at this time were similar to other post- Sunday values, as were the charge currents At this voltage level, the auxiliary battery was considerably undercharged. We appeared to be in a stable, though not fully nominal mode, and I was in the process of noting the status of things to the command team to decide on the next course of action when, at 00:39:59 utc the battery voltage suddenly dropped from its post-Sunday level of 25.5 to 26.5 volts, down to 18 volts. This was accompanied by an expected increase in charge current. The safety software shut off the S2 Tx at the next MA change, and I brought it back up manually twice, noting that the voltage was relatively stable at about 14 volts with heavy charge current. After letting things sit for a few minutes, a third attempt to start the S2 Tx was not successful.

Our current best understanding is that we suffered a catastrophic failure of the main battery which is clamping the buss voltage at a low level. Accordingly, we have been concentrating our efforts on trying to connect the auxiliary battery to the main buss and disconnect the main battery, placing it on trickle charge for further analysis. If we can manage just 10 volts on the main buss to activate the IHU-1 we should be able to accomplish this with the simple machine codes noted above, and the voltages should come up to levels where the S2 will function.

If you examine the telemetry during this time, 10 minutes after the ow-voltage "event", the temperature of heat pipe 4 + X + Y takes a 34.2 deg.C apparent increase in temperature over a span of 14 seconds (1 block interval), and continues to rapidly increase almost to the top of the scale. Given the timing of this change, 10 minutes after the low-voltage "event", its rapidity, the thermal inertia of the large, circumferential heat pipes, etc... it is difficult to believe that this is real. Most importantly, this heat pipe sensor sits in a "benign" area at the apex of panels 1 and 6, near the high-gain end of the spacecraft. There are no batteries or BCR's in this region, only the controller for the liquid rocket motor (which was powered off!), and the L1 receiver, which shows no accompanying change in temperature. All of the BCR's and batteries are much closer to heat pipes 1 and 2 at the omni end of the spacecraft. Therefore, this may represent a failure of the software/hardware due to low voltage, rather than a real temperature increase. However, this will be studied further. No other telemetry anomalies have been noted but we continue to look.

-W4SM for the AO-40 Command Team
 

26 Jan 21:39 UTC    We had an extreme low voltage event on the last orbit. This shut off the S2 Tx, and probably crashed the IHU-1. At the moment the main and aux. batteries have been tied together and should be charging. We will wait until the next orbit to see if the voltage on the batteries comes up to speed and we can restart the IHU-1 and reload software. ...fingers crossed.

-W4SM for the AO-40 Command Team
 

25. Jan 21:43 UTC   On orbit 1486, the main battery voltage ran lower than expected. This was due in part to usage and high solar angle as mentioned previously, but even after the passbands were shut off automatically the voltages did not come back up as expected, in spite of apparently positive battery charge currents. At the beginning of the eclipse on orbit 1486 at MA =225, the battery voltage rapidly dropped just below the S2 Tx cutoff voltage of 24.0 volts and the S2 Tx shut down. to protect the battery and IHU's. The eclipse ended at about MA=237, approx. 55 minutes later. We expect the S2 Tx to start back up as programmed at MA =2 (as I type this), though it has been known to be "balky" before on restarts and could require manual cycling.
 

The command team is investigating this anomaly to determine if the main battery may have developed a single bad (shorted) cell or if there is a problem with the battery set voltage, solar array set voltages, or other problem. Until we sort this out the passbands will be off for one or more orbits. If the battery charge or array set points are wrong, they can be changed. If we have one bad ( shorted ) cell we can reset charge/trigger points to account for this, and we have the completely redundant auxiliary battery, if needed. We hope to be able to work through this and correct things without too much down time. We'll update you here as soon as we know more.

-W4SM for the AO-40 Command Team