Generated for ASP.NET Core migration planning
The DDEC calibration processing system handles downloading, parsing, encrypting/decrypting, compressing/decompressing, and releasing engine calibration data files. This document details every component involved.
Source: src/ddec-web/src/main/java/com/detroitdiesel/ddec/engineering/V2DdecCalibrationSplitter.java
Parses the binary download.dat file (produced by the batch job) into individual calibration records. The file contains mixed EBCDIC-encoded headers and binary content.
| Record Type | Description | Content Type | Has Length in Header |
|---|---|---|---|
| D01 | DDEC2 Master Unit | DDEC2_BINARY | No |
| D02 | DDEC2 Master Test Cal | DDEC2_BINARY | No |
| D03 | DDEC2 Historical Unit | DDEC2_BINARY | No |
| D04 | DDEC2 Historical Test Cal | DDEC2_BINARY | No |
| D08 | DDEC III/IV Base Cal | COMPRESSED_BINARY | Yes |
| D09 | DRS Screen Message | ASCII | No (always 47 chars) |
| D10 | DDEC III/IV Master Prod Cal | COMPRESSED_BINARY | Yes |
| D11 | DDEC III/IV Master Prod Unit | ASCII | No |
| D12 | DDEC III/IV Receiver 2 Prod Unit | ASCII | No |
| D13 | DDEC III/IV Receiver 3 Prod Unit | ASCII | No |
| D14 | DDEC III/IV Master Test Cal | COMPRESSED_BINARY | Yes |
| D15 | DDEC III/IV Receiver 2 Test Cal | COMPRESSED_BINARY | Yes |
| D16 | DDEC III/IV Receiver 3 Test Cal | COMPRESSED_BINARY | Yes |
| D20 | DDEC III Split Base Cal | UNCOMPRESSED_ENCRYPTED_BINARY | Yes |
| D28 | Split Historical Base Cal | UNCOMPRESSED_ENCRYPTED_BINARY | Yes |
| D51 | DDEC V Master Prod Unit | ASCII | No |
| D54 | DDEC V Master Test Cal | COMPRESSED_BINARY | Yes |
| D58 | DDEC V Base Cal | COMPRESSED_BINARY | Yes |
| D80 | Pending Historical Base Cal | COMPRESSED_BINARY | Yes |
| D88 | Historical Base Cal | COMPRESSED_BINARY | Yes |
Method 1: From Header (Most Common)
For records with headerContainsNumBytes() == true:
Header: "D0806N04D0602 000000000000004079"
Bytes 0-2: "D08" (record type)
Bytes 3-20: "06N04D0602 " (calibration name)
Bytes 21-36: " 000000000000004079" (padded numeric length)
Parsing: Strip non-digits, parse as integer -> 4079 bytes of content
Method 2: Calculated (ASCII Records)
For D09, D11, D12, D13, D51:
contentLength = nextRecordOffset - (currentOffset + recordTypeLength + calNameLength)
Special case: D09 is always 47 characters
Method 3: DDEC2 Calculation
For D01-D04: DDEC2 has 16 extra bytes of timestamp before binary content
contentLength = nextRecordOffset - currentOffset
// Record type definition
public record DdecDataRecordType(
String recType, // "D08"
String shortDescription, // "DDECIII_BASE_CAL"
String longDescription, // "DDECIV/III Base Cal Record"
String typeOfFileContent, // "COMPRESSED_BINARY"
boolean headerContainsNumBytes // true
)
// Parsed header record
public record DdecHeaderContentRecord(
String recTypeIndicator, // "D08"
String calName, // "06N04D0602"
String calibrationType, // "DDECIII_BASE_CAL : DDECIV/III Base Cal Record"
String asciiHeaderContent, // Full header as ASCII string
Integer contentLength, // 4079
DdecDataRecordType calTypeInfo // Reference to type definition
)
// Extracted file record
public record DdecExtractedFileRecord(
String fileName, // Full path to extracted file
DdecDataRecordType calTypeInfo // Type information
)
Source: src/ddec-web/src/main/java/com/detroitdiesel/ddec/engineering/util/EncodingUtil.java
The algorithm uses a simple two-step process for encryption/decryption.
Step 1: XOR with Key
private static int applyKey(int byteIn) {
int keyMask = 63; // 0x3F = 00111111
byteIn ^= keyMask;
return byteIn & 0xff;
}
Step 2: Nibble Swap
private static int arrangeByte(int byteIn) {
int switchMask = 15; // 0x0F
int saveByte = (byteIn & switchMask); // Get low nibble
saveByte = (saveByte << 4); // Shift to high position
int tempByte = (byteIn >> 4); // Get high nibble
int result = (tempByte | saveByte); // Combine swapped
return result & 0xff;
}
Example:
Input byte: 10110011 (0xB3)
After XOR: 10001100 (0x8C) [0xB3 XOR 0x3F]
Low nibble: 00001100 -> shift left -> 11000000
High nibble: 00001000
After swap: 11001000 (0xC8)
Encryption reverses the operation order - nibble swap FIRST, then XOR:
private static int[] encrypt(byte[] fileBytes) {
int[] encryptedFileBytes = new int[fileBytes.length];
for (int i = 0; i < fileBytes.length; i++) {
int initialValue = fileBytes[i] & 0xFF; // Ensure unsigned
int saveByte = arrangeByte(initialValue); // Swap nibbles FIRST
int finalByte = applyKey(saveByte); // Then XOR
encryptedFileBytes[i] = finalByte;
}
return encryptedFileBytes;
}
Java: Uses IBM Code Page 037 (Cp037)
private static final String CP_EBCDIC_TO_ASCII = "Cp037";
String ascii = new String(ebcdicBytes, CP_EBCDIC_TO_ASCII);
byte[] ebcdic = asciiString.getBytes(CP_EBCDIC_TO_ASCII);
C# Equivalent:
Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
var ebcdic = Encoding.GetEncoding(37); // IBM037
string ascii = ebcdic.GetString(ebcdicBytes);
byte[] ebcdicBytes = ebcdic.GetBytes(asciiString);
Source: src/ddec-web/src/main/java/com/detroitdiesel/ddec/engineering/JavaToC.java
Uses PKWARE DCL Implode compression algorithm via JNI:
XCompress.dllXCompress.so/opt/app/clib/private native int init(String logPath, int logFlag);
private native int implode(String fileName); // Compress
private native int explode(String fileName); // Decompress
JavaToC xCompress = new JavaToC(baseDir, true);
// Decompress: creates {fileName}-exp
xCompress.decompressFile(fileName);
// Compress: creates {fileName}-imp
xCompress.compressFile(fileName);
[DllImport("XCompress.so", CallingConvention = CallingConvention.Cdecl)]
private static extern int explode(string fileName);
Source: src/ddec-web/src/main/java/com/detroitdiesel/ddec/engineering/util/R24kd500shDownloadRequestBuilder.java
| Parameter | Description |
|---|---|
| 100 | Engine Serial Download |
| 200 | Base Cal Without History |
| 600 | Base Cal With History |
if (calibrationName.startsWith("06N04") && withHistory) {
return "600";
} else if (calibrationName.startsWith("06N04")) {
return "200";
}
return calibrationType; // Default: use provided type
DDECREQ0.DAT (request0.file):
{CalibrationName}{UserID} {Padding}
Example: 06N04D0602ADMIN 9999999999
DDECHIST.DAT (history.file):
H99 {UserID}{Padding}{DateTime}
Example: H99 ADMIN999999999901282024153045123
Date format: MMddyyyyHHmmssSSS
User Job Script ({userId}500.sh):
#!/bin/ksh
. ${GLOBAL_SCRIPT}
XXXXX=ADMIN
YYY=600
. ${SCRIPT_DIR}/batch/r24kd500.sh
Files uploaded via SFTP, then polls for result:
Processing flow for each extracted record:
| Content Type | Processing Steps |
|---|---|
| COMPRESSED_BINARY | Decompress -> Decrypt -> Add to ZIP |
| UNCOMPRESSED_ENCRYPTED_BINARY | Decrypt only -> Add to ZIP |
| ASCII / DDEC2 | No processing -> Add to ZIP |
ZIP naming: split_{filename}_{userId}_{calName}_jobparam_{param}_{timestamp}.zip
Files included:
download.file - Original raw filedownload.file.ascii-troubleshooting - Full EBCDIC to ASCII conversionSource: src/ddec-web/src/main/java/com/detroitdiesel/ddec/engineering/util/R24kd505shReleaseRequestBuilder.java
| Parameter | Description |
|---|---|
| 300 | Base Calibration Release |
| 400 | Test Calibration Release |
Variable-length records with mixed EBCDIC and binary content:
[Record 1: D08 - Base Calibration]
[Record 2: D09 - Screen Message]
[Record 3: D11 - Unit Parameters]
[Record 4: D88 - Historical Calibration]
...
+----------------+------------------+------------------+
| Record Type | Calibration Name | Remaining Header |
| 3 bytes EBCDIC | 18 bytes EBCDIC | Variable EBCDIC |
+----------------+------------------+------------------+
| File Content |
| Variable length (ASCII or Binary) |
+------------------------------------------------------+
D08 (Compressed Binary Base Calibration):
Offset Length Content
0-2 3 "D08" (EBCDIC)
3-20 18 Calibration name (EBCDIC, padded)
21-28 8 Create date "20240128" (EBCDIC)
29-34 6 Create time "153045" (EBCDIC)
35-52 18 Padded spaces (EBCDIC)
53-? ? Numeric length as string (EBCDIC)
?-end N Compressed + encrypted binary data
D09 (ASCII Screen Message):
Offset Length Content
0-2 3 "D09" (EBCDIC)
3-20 18 Calibration name (EBCDIC, padded)
21-67 47 Message text (EBCDIC)
D11 (ASCII Unit Parameters):
Offset Length Content
0-2 3 "D11" (EBCDIC)
3-20 18 Engine serial number (EBCDIC)
21-end var ASCII unit parameter data (EBCDIC encoded)
D01 (DDEC2 Uncompressed Binary):
Offset Length Content
0-2 3 "D01" (EBCDIC)
3-20 18 Engine serial + model (EBCDIC)
21-36 16 Timestamp (EBCDIC)
37-end var Uncompressed binary calibration data
split_download.file_ADMIN_06N04D0602_jobparam_600_1706387200000.zip
├── download.file (Raw)
├── download.file.ascii-troubleshooting (Debug)
├── download.file_file_1_06N04D0602_D08_DDECIII_BASE_CAL_...dat
├── download.file_file_1_06N04D0602_D08_DDECIII_BASE_CAL_...dat-exp
├── download.file_file_1_...dat-exp.decrypted (Final)
├── download.file_file_2_06N04D0602_D09_DRS_SCREEN_MESSAGE_...dat
└── download.file_file_3_06N04D0602_D88_HIST_BASE_CAL_...dat-exp.decrypted
File Suffixes:
.dat - Extracted (raw).dat-exp - Decompressed (exploded).dat-exp.decrypted - Decompressed and decrypted.dat.decrypted - Decrypted only (no compression)Key Steps:
| Step | Program | Purpose |
|---|---|---|
| JS010 | IDCAMS | Delete old files |
| JS020 | WaapInit | Initialize temp datasets |
| JS025 | WaapDsut | Backup history file |
| JS030 | R24PUH01 | Update history (IMS BMP) |
| JS040 | R24PRC01 | Core: Process calibration request (IMS BMP) |
| JS070 | WaapDsut | Copy grief file |
| JS080 | WaapDsut | Create download.file from temp output |
| JS090 | WaapInit | Error handling - create empty file on failure |
JS040 Detail (Core Step):
assignFile R35DGIV1 "${XXXXX}/r24at/request0.file" -d "OLD"
assignFile R35DGOV1 "${XXXXX}/r24at/d24301d1.file" -d "MOD"
assignFile R35DGO02 "${XXXXX}/r24at/d24301d4.file" -d "MOD"
assignFile DATERANG "pddad/syscm/prod/sysin/r24irc01.file"
# Append the YYY parameter (100/200/600)
addLine DATERANG "${YYY}..."
callIms1Bmp -mbr "R24PRC01" -psb "R24PRC08"
Functions Based on YYY Parameter:
| YYY | Function |
|---|---|
| 1xx | Create new unit calibration |
| 2xx | Retrieve base calibration (no history) |
| 3xx | Insert/upload base calibration |
| 4xx | Release test calibration |
| 6xx | Retrieve calibration with full history |
Operations:
| Component | Package/Library |
|---|---|
| Web Framework | ASP.NET Core 8.0 |
| EBCDIC Support | System.Text.Encoding.CodePages (NuGet) |
| SFTP Client | SSH.NET / Renci.SshNet (NuGet) |
| ZIP Creation | System.IO.Compression |
| Compression | P/Invoke to XCompress.so or pure C# implementation |
namespace DDEC.Engineering
{
// Core processing
public class CalibrationSplitter { }
public class EncodingUtility { }
public class CompressionService { }
// Request builders (can be simplified to direct DB calls)
public class CalibrationDownloadService { }
public class CalibrationReleaseService { }
// Models
public record DdecDataRecordType;
public record DdecHeaderContentRecord;
public record DdecExtractedFileRecord;
// Controller
[ApiController]
[Route("api/engineering")]
public class EngineeringController : ControllerBase { }
}
System.Text.Encoding.CodePages NuGet package