The calibration download/release pipeline - the most complex non-IMS subsystem
This is the ONLY part of the application that does NOT use the IMS/DLI emulation framework
The Engineering subsystem handles calibration file download and release (upload) for DDEC engine ECMs. It is the one area where DXC wrote new Java code rather than converting COBOL, because "no COBOL code existed for this functionality."
Browser (JSP pages)
│
▼
EngineeringController (REST API)
│
├── Download Flow:
│ ├── R24kd500shDownloadRequestBuilder
│ │ ├── Builds request0.file, history.file, <user>500.sh
│ │ ├── SFTPs files to batch server
│ │ └── Polls for download.file response
│ │
│ ├── V2DdecCalibrationSplitter
│ │ ├── Parses download.dat (EBCDIC binary)
│ │ ├── Identifies 16+ record types (D01-D88)
│ │ ├── Extracts data record segments
│ │ └── Per-segment processing:
│ │ ├── ASCII: EBCDIC→ASCII conversion
│ │ ├── Compressed binary: XCompress→Decrypt
│ │ └── Uncompressed encrypted: Decrypt only
│ │
│ ├── JavaToC (XCompress JNI wrapper)
│ │ └── Calls native XCompress.so/.dll for DCL Implode decompression
│ │
│ └── EncodingUtil (XOR encryption/decryption)
│ ├── applyKey(): XOR with keyMask=63 (0x3F)
│ └── arrangeByte(): nibble swap (low 4 bits ↔ high 4 bits)
│
└── Release Flow:
└── R24kd505shReleaseRequestBuilder
├── Accepts request0.file from DRS client
├── Builds history.file, <user>505.sh
├── SFTPs to batch server
└── Polls for response
File: engineering/util/EngineeringController.java
Security: @PreAuthorize("hasRole('ANY')")
| Method | Path | Function |
|---|---|---|
| GET | /engineering/ddec/basecal | Returns download-calibration.jsp page |
| POST | /engineering/ddec/basecal/process | Executes download + split pipeline |
| POST | /engineering/ddec/releasecal | Executes release pipeline |
QreContext.getImsSession().getUserId()R24kd500shDownloadRequestBuilder.buildRequiredFilesAndRunJob()V2DdecCalibrationSplitter.processDownloadDatFile()archiveFileName.startsWith("split_download.file_" + userId) to prevent LFIR24kd505shReleaseRequestBuilder.releaseCalibration()File: engineering/util/EncodingUtil.java
Migration Complexity: Simple
The encryption is a simple XOR + nibble swap. Decrypt and encrypt use the same operations but in reverse order.
Decrypt: applyKey(byte) → arrangeByte(byte)
Encrypt: arrangeByte(byte) → applyKey(byte)
Input: byte
Output: byte XOR 0x3F (63 decimal)
Example: 0xA5 XOR 0x3F = 0x9A
Input: byte (e.g., 0xA5 = 1010 0101)
Output: swap low 4 bits and high 4 bits
Steps:
1. saveByte = byte AND 0x0F → 0x05 (low nibble)
2. saveByte = saveByte << 4 → 0x50 (shift to high)
3. tempByte = byte >> 4 → 0x0A (high nibble shifted to low)
4. result = tempByte OR saveByte → 0x5A
So 0xA5 becomes 0x5A (nibbles swapped)
Encrypt 0xA5:
arrangeByte(0xA5) = 0x5A (nibble swap)
applyKey(0x5A) = 0x65 (XOR 0x3F)
→ Encrypted: 0x65
Decrypt 0x65:
applyKey(0x65) = 0x5A (XOR 0x3F)
arrangeByte(0x5A) = 0xA5 (nibble swap back)
→ Decrypted: 0xA5
public static class EncodingUtil
{
private const int KeyMask = 0x3F; // 63
private const int SwitchMask = 0x0F; // 15
public static byte[] Decrypt(byte[] data)
{
var result = new byte[data.Length];
for (int i = 0; i < data.Length; i++)
{
int b = ApplyKey(data[i] & 0xFF);
result[i] = (byte)ArrangeByte(b);
}
return result;
}
public static byte[] Encrypt(byte[] data)
{
var result = new byte[data.Length];
for (int i = 0; i < data.Length; i++)
{
int b = ArrangeByte(data[i] & 0xFF);
result[i] = (byte)ApplyKey(b);
}
return result;
}
private static int ApplyKey(int b) => (b ^ KeyMask) & 0xFF;
private static int ArrangeByte(int b) =>
(((b & SwitchMask) << 4) | (b >> 4)) & 0xFF;
}
File: engineering/V2DdecCalibrationSplitter.java
Migration Complexity: Complex (binary parsing, EBCDIC, multi-format)
| Code | Short Name | Content Type | Header Has Length? | Description |
|---|---|---|---|---|
| D01 | DDECII_MASTER_1_ECM_CAL | DDEC2_UNCOMPRESSED_BINARY | No | DDEC II Master Production unit |
| D02 | DDECII_RECEIVER_2_ECM_CAL | DDEC2_UNCOMPRESSED_BINARY | No | DDEC II Receiver 1 Production |
| D03 | DDECII_RECEIVER_3_ECM_CAL | DDEC2_UNCOMPRESSED_BINARY | No | DDEC II Receiver 2 Production |
| D04 | DDECII_BASE_CAL | DDEC2_UNCOMPRESSED_BINARY | No | DDEC II Base Cal |
| D08 | DDECIII_BASE_CAL | COMPRESSED_BINARY | Yes | DDEC III/IV Base Cal |
| D09 | DRS_SCREEN_MESSAGE | ASCII | No | Screen Message (47 chars) |
| D11 | DDECIII_MASTER_1_PROD_UNIT | ASCII | No | DDEC III/IV Master Prod Unit |
| D12 | DDECIII_RECEIVER_2_PROD_UNIT | ASCII | No | DDEC III/IV Receiver 1 Prod |
| D13 | DDECIII_RECEIVER_3_PROD_UNIT | ASCII | No | DDEC III/IV Receiver 2 Prod |
| D14 | DDECIII_MASTER_1_TEST_CAL | COMPRESSED_BINARY | Yes | DDEC III/IV Master Test |
| D15 | DDECIII_RECEIVER_2_TEST_CAL | COMPRESSED_BINARY | Yes | DDEC III/IV Receiver 1 Test |
| D16 | DDECIII_RECEIVER_TEST_CAL | COMPRESSED_BINARY | Yes | DDEC III/IV Receiver 2 Test |
| D20 | DDECIII_SPLIT_BASE_CAL | UNCOMPRESSED_ENCRYPTED_BINARY | Yes | Split Cal Record |
| D28 | SPLIT_HIST_BASE_CAL | UNCOMPRESSED_ENCRYPTED_BINARY | Yes | Historical Split Cal |
| D51 | DDECV_MASTER_PROD_UNIT | ASCII | No | DDEC V Master Prod Unit |
| D54 | DDECV_MASTER_TEST_CAL | COMPRESSED_BINARY | Yes | DDEC V Master Test |
| D58 | DDECV_BASE_CAL | COMPRESSED_BINARY | Yes | DDEC V Base Cal |
| D80 | PEND_HIST_BASE_CAL | COMPRESSED_BINARY | Yes | Pending Historical Cal |
| D88 | HIST_BASE_CAL | COMPRESSED_BINARY | Yes | Historical Base Cal |
[D08][06N04D0602 ][000000000000004079][<4079 bytes of compressed encrypted binary>]
│ │ │ └─ Content (binary or ASCII)
│ │ └─ Remaining header (content length for binary types)
│ └─ Calibration name (18 chars EBCDIC, space-padded)
└─ Record type (3 chars EBCDIC)
Format Variations by Type:
[Dxx][18 chars serial+model][16 char date][uncompressed binary][Dxx][18 chars 06N04+spaces][length][compressed encrypted binary][Dxx][18 chars serial+model][plain ASCII][Dxx][18 chars serial+model][length][compressed encrypted binary][D09][15 chars name+spaces][47 chars ASCII message]download.dat (EBCDIC binary blob)
│
├─ 1. Convert entire file EBCDIC→ASCII for offset detection
├─ 2. Find all positions of calibrationName in ASCII version
├─ 3. Sort offsets to determine segment boundaries
│
└─ For each segment:
├─ 4. Parse header (3 bytes type + 18 bytes name + remaining)
├─ 5. Determine content length (from header or calculated)
├─ 6. Read content bytes
│
└─ Based on content type:
├─ ASCII: Convert EBCDIC→ASCII, write to file
├─ COMPRESSED_BINARY: Write to file → XCompress.explode() → EncodingUtil.decrypt()
├─ UNCOMPRESSED_ENCRYPTED_BINARY: Write to file → EncodingUtil.decrypt()
└─ DDEC2_UNCOMPRESSED_BINARY: Write to file as-is
A ZIP file containing:
File: engineering/JavaToC.java
Migration Complexity: High (native library dependency)
private native int init(String logPath, int logFlag);
private native int implode(String fileName); // Compress
private native int explode(String fileName); // Decompress
/opt/app/clib/XCompress.soC:/opt/app/clib/XCompress.dllSystem.load() in static initializer<original>-imp<original>-exp// Option 1: P/Invoke
public static class XCompress
{
[DllImport("XCompress", CallingConvention = CallingConvention.Cdecl)]
private static extern int init(string logPath, int logFlag);
[DllImport("XCompress", CallingConvention = CallingConvention.Cdecl)]
private static extern int explode(string fileName);
[DllImport("XCompress", CallingConvention = CallingConvention.Cdecl)]
private static extern int implode(string fileName);
}
// Option 2: Pure C# (preferred if source/spec available)
// PKWARE DCL Implode is documented:
// http://fileformats.archiveteam.org/wiki/PKWARE_DCL_Implode
File: engineering/util/R24kd500shDownloadRequestBuilder.java
Migration Complexity: Medium (SFTP + batch job orchestration)
1. Determine job parameter (YYY):
- "100" = Download Engine Serial Number
- "200" = Download Calibration Without History
- "600" = Download Calibration With History
2. Build request files locally:
- request0.file: "{calName}{USERID} 9999999999"
- history.file: "H99 {USERID}9999999999{MMddyyyyHHmmssSSS}" (or uploaded file)
- {userid}500.sh: Korn shell script
3. SFTP all files to batch server:
- Host: STNAFDDCL1628 (prod) / DTNAFDDCL1625 (dev)
- User: ddecwebapp
- Directory: /opt/qte/data/{USERID}/r24at/
4. Poll for download.file (retry with timeout)
5. Download download.file via SFTP
6. Delete remote download.file after download
7. Interpret results:
- File > 100 bytes = success
- File ≤ 100 bytes = error message (EBCDIC→ASCII)
- No file = error
#!/bin/ksh
. ${GLOBAL_SCRIPT}
XXXXX={USERID}
YYY={PARAMETER}
. ${SCRIPT_DIR}/batch/r24kd500.sh
Local: /opt/qte/data/{USERID}/r24at/
Remote: /opt/qte/data/{USERID}/r24at/ (same path on SFTP server)
Batch: /opt/qte/scripts/batch/drs/{userid}500.sh
File: engineering/util/R24kd505shReleaseRequestBuilder.java
Migration Complexity: Medium (nearly identical to download)
Same pattern as download but:
record DdecDataRecordType(
String recType, // "D08", "D58", etc.
String shortDescription, // "DDECIII_BASE_CAL"
String longDescription, // "DDECIV/III Base Cal Record"
String typeOfFileContent, // "ASCII", "COMPRESSED_BINARY", etc.
boolean headerContainsNumBytes // true if header includes content length
)
record DdecHeaderContentRecord(
String recTypeIndicator, // "D08"
String calName, // "06N04D0602"
String calibrationType, // "DDECIII_BASE_CAL : DDECIV/III Base Cal Record"
String asciiHeaderContent, // Full header as ASCII
Integer contentLength, // Bytes of content
DdecDataRecordType calTypeInfo // Reference to type definition
)
record DownloadCalibrationFileRecord(
String calibrationName, // "06N04D0602" or engine serial
String calibrationType, // Job parameter override
MultipartFile historyFile // Optional uploaded history file
)
record ReleaseCalibrationFileRecord(
MultipartFile reqFile0, // Uploaded request0.file from DRS
String calibrationType // Job parameter (300=prod, 400=test)
)
class EngineeringBatchJobResponse {
String status; // "OK" or "ERROR"
String message; // Human-readable result
String downloadFileName; // download.file (if success)
String archiveFileName; // ZIP file (after split)
String jobParameter; // "200", "600", etc.
}
BASE_DIRECTORY = "/opt/qte/data/"
BASE_DIRECTORY_WINDOWS = "c:/opt/qte/data/"
SUB_DIRECTORY = "/r24at/"
TEMP_DIR = "tmp/"
DOWNLOAD_FILE = "download.file"
IDENT_06N04 = "06N04"
CP_EBDDIC_TO_ASCII = "Cp037"
C_LIB_PATH = "/opt/app/clib/"
Direct port of XOR + nibble swap. Already shown above. Include unit tests with round-trip verification.
public class CalibrationSplitter
{
// Use System.Text.Encoding.GetEncoding(37) for EBCDIC
private static readonly Encoding Ebcdic = Encoding.GetEncoding(37);
public string ProcessDownloadFile(string baseDir, string filename,
string calibrationName, string userId, string jobParameter)
{
// 1. Read entire file
// 2. Convert to ASCII for offset detection
// 3. Find all calibration name positions
// 4. Parse each segment header
// 5. Extract and process content by type
// 6. Build ZIP using System.IO.Compression.ZipArchive
}
}
Options:
In the new architecture, the batch job can be replaced with direct database operations:
Current: Browser → REST API → Build files → SFTP → Batch server → KSH script → Database → SFTP response → Parse
New: Browser → REST API → Direct DB query → Return calibration data → Parse → Return ZIP
This eliminates:
The database queries that the KSH script executes can be run directly from the ASP.NET Core application against SQL Server.
System.Text.Encoding.GetEncoding(37) - requires NuGet package System.Text.Encoding.CodePages/opt/qte/data/ - need configuration-driven pathsDateTimeOffset.ToString() in C#.