Downloads hanging- Transient HTTP error

W
WestWUN ID: 8508159 Posts: 3
30 Oct 2023 12:06 PM

I'm new to the world of Charity Engine and I appear to have an issue.  I'll check my transfer tab and see tasks failing to download.  Selecting retry will work, but it will normally take several tries to initiate the download.  I have searched several threads and added the download servers (download1.charityengine.com / download2.charityengine.com / download3.charityengine.com) to my exceptions in my firewall with no change.  Additionally, I am seeing "transient http error" in my logs.  Any suggestions on how to get this working correctly?

10/30/2023 6:20:48 AM | CharityEngine | Temporarily failed download of input.fasta_32799ddd9fc58e34be850b9b24088363: transient HTTP error

10/30/2023 6:20:48 AM | CharityEngine | Backing off 00:05:12 on download of input.fasta_32799ddd9fc58e34be850b9b24088363

10/30/2023 6:20:52 AM | CharityEngine | Started download of input.fasta_32799ddd9fc58e34be850b9b24088363

10/30/2023 6:20:54 AM | CharityEngine | Temporarily failed download of input.fasta_32799ddd9fc58e34be850b9b24088363: transient HTTP error

10/30/2023 6:20:54 AM | CharityEngine | Backing off 00:15:14 on download of input.fasta_32799ddd9fc58e34be850b9b24088363

10/30/2023 6:20:55 AM | CharityEngine | Started download of input.fasta_32799ddd9fc58e34be850b9b24088363

10/30/2023 6:20:58 AM | CharityEngine | Temporarily failed download of input.fasta_32799ddd9fc58e34be850b9b24088363: transient HTTP error

10/30/2023 6:20:58 AM | CharityEngine | Backing off 00:20:03 on download of input.fasta_32799ddd9fc58e34be850b9b24088363

10/30/2023 6:24:37 AM | CharityEngine | Started download of input.fasta_32799ddd9fc58e34be850b9b24088363

10/30/2023 6:24:40 AM | CharityEngine | Temporarily failed download of input.fasta_32799ddd9fc58e34be850b9b24088363: transient HTTP error

10/30/2023 6:24:40 AM | CharityEngine | Backing off 00:46:20 on download of input.fasta_32799ddd9fc58e34be850b9b24088363

10/30/2023 6:24:42 AM | CharityEngine | Started download of input.fasta_32799ddd9fc58e34be850b9b24088363

10/30/2023 6:24:44 AM | CharityEngine | Temporarily failed download of input.fasta_32799ddd9fc58e34be850b9b24088363: transient HTTP error

10/30/2023 6:24:44 AM | CharityEngine | Backing off 01:36:22 on download of input.fasta_32799ddd9fc58e34be850b9b24088363

10/30/2023 6:24:46 AM | CharityEngine | Started download of input.fasta_32799ddd9fc58e34be850b9b24088363

10/30/2023 6:24:48 AM | CharityEngine | Temporarily failed download of input.fasta_32799ddd9fc58e34be850b9b24088363: transient HTTP error

10/30/2023 6:24:48 AM | CharityEngine | Backing off 03:50:36 on download of input.fasta_32799ddd9fc58e34be850b9b24088363

10/30/2023 6:24:50 AM | CharityEngine | Started download of input.fasta_32799ddd9fc58e34be850b9b24088363

10/30/2023 6:24:52 AM | CharityEngine | Temporarily failed download of input.fasta_32799ddd9fc58e34be850b9b24088363: transient HTTP error

10/30/2023 6:24:52 AM | CharityEngine | Backing off 05:28:50 on download of input.fasta_32799ddd9fc58e34be850b9b24088363

10/30/2023 6:24:54 AM | CharityEngine | Started download of input.fasta_32799ddd9fc58e34be850b9b24088363

10/30/2023 6:24:57 AM | CharityEngine | Finished download of input.fasta_32799ddd9fc58e34be850b9b24088363

10/30/2023 6:24:58 AM | CharityEngine | Starting task b1fb9c40a531031f8863d907f5b8b469_1_48_0

10/30/2023 6:24:58 AM | CharityEngine | Starting task 81fef2f418f10f6d8cc1e07507afd2a7_1_48_0

10/30/2023 6:25:11 AM | Rosetta@home | update requested by user

10/30/2023 6:25:12 AM | Rosetta@home | Sending scheduler request: Requested by user.

10/30/2023 6:25:12 AM | Rosetta@home | Not requesting tasks: don't need (CPU: not highest priority project; NVIDIA GPU: )

10/30/2023 6:25:14 AM | Rosetta@home | Scheduler request completed

10/30/2023 6:25:19 AM | CharityEngine | Sending scheduler request: To fetch work.

10/30/2023 6:25:19 AM | CharityEngine | Requesting new tasks for CPU and NVIDIA GPU

10/30/2023 6:25:20 AM | Charity Engine | Scheduler request completed: got 1 new tasks

10/30/2023 6:25:21 AM | Charity Engine | Computation for task 81fef2f418f10f6d8cc1e07507afd2a7_1_48_0 finished

10/30/2023 6:25:22 AM | Charity Engine | Started download of input.fasta_cbebd91e4472752b3db371d5425db0ac

10/30/2023 6:25:23 AM | Charity Engine | Started upload of 81fef2f418f10f6d8cc1e07507afd2a7_1_48_0_r803932021_0

10/30/2023 6:25:24 AM | Charity Engine | Finished upload of 81fef2f418f10f6d8cc1e07507afd2a7_1_48_0_r803932021_0

10/30/2023 6:25:25 AM | Charity Engine | Temporarily failed download of input.fasta_cbebd91e4472752b3db371d5425db0ac: transient HTTP error

10/30/2023 6:25:25 AM | Charity Engine | Backing off 00:03:07 on download of input.fasta_cbebd91e4472752b3db371d5425db0ac

10/30/2023 6:25:30 AM | Charity Engine | Computation for task b1fb9c40a531031f8863d907f5b8b469_1_48_0 finished

 
Matt ID: 44 Posts: 290
01 Nov 2023 12:02 AM

That was a data congestion issue; should have resolved itself. (*It wouldn't be expected to interfere with other Charity Engine work on your device - or with anything else; the log is just a comprensive record of activity.). But let us know if the issue persists, or if there are any symptoms other than the log entries.

Matt ID: 44 Posts: 290
01 Nov 2023 12:02 AM

That was a data congestion issue; should have resolved itself. (*It wouldn't be expected to interfere with other Charity Engine work on your device - or with anything else; the log is just a comprensive record of activity.). But let us know if the issue persists, or if there are any symptoms other than the log entries.

W
WestWUN ID: 8508159 Posts: 3
01 Nov 2023 12:34 AM

Will do. Thanks for the help.

Кирилл Фролов ID: 3765769 Posts: 12
03 Nov 2023 12:22 PM

I have reset project and it started stalling on ce11 downloads.
Can you increase network capacity on the servers?

W
WestWUN ID: 8508159 Posts: 3
03 Nov 2023 12:42 PM

The download issues are continuing. It seems that most times when I check my transfer tab, the CE22 tasks are hung.  The status block will report "Backing off of download for "x" amount of time".  I will then "retry" the download with success after 1-10+ attempts.

In addition, I am also getting some failed downloads reflecting checksum errors. Below is a snapshot of what my event logs are showing concerning that.

11/3/2023 7:24:17 AM | Charity Engine | Started download of input.fasta_30aad8fe45dc2e0a76913a0d844ece6b

11/3/2023 7:24:20 AM | Charity Engine | Temporarily failed download of input.fasta_30aad8fe45dc2e0a76913a0d844ece6b: transient HTTP error

11/3/2023 7:24:20 AM | Charity Engine | Backing off 00:07:45 on download of input.fasta_30aad8fe45dc2e0a76913a0d844ece6b

11/3/2023 7:32:06 AM | Charity Engine | Started download of input.fasta_30aad8fe45dc2e0a76913a0d844ece6b

11/3/2023 7:32:09 AM | Charity Engine | Temporarily failed download of input.fasta_30aad8fe45dc2e0a76913a0d844ece6b: transient HTTP error

11/3/2023 7:32:09 AM | Charity Engine | Backing off 00:08:21 on download of input.fasta_30aad8fe45dc2e0a76913a0d844ece6b

11/3/2023 7:32:13 AM | Charity Engine | Sending scheduler request: Requested by project.

11/3/2023 7:32:13 AM | Charity Engine | Not requesting tasks: some download is stalled

11/3/2023 7:32:15 AM | Charity Engine | Scheduler request completed

11/3/2023 7:34:32 AM | Charity Engine | Started download of input.fasta_30aad8fe45dc2e0a76913a0d844ece6b

11/3/2023 7:34:38 AM | Charity Engine | Finished download of input.fasta_30aad8fe45dc2e0a76913a0d844ece6b

11/3/2023 7:34:38 AM | Charity Engine | [error] File input.fasta_30aad8fe45dc2e0a76913a0d844ece6b has wrong size: expected 27056938, got 26771456

11/3/2023 7:34:38 AM | Charity Engine | [error] Checksum or signature error for input.fasta_30aad8fe45dc2e0a76913a0d844ece6b