MR2 Owners Club Forum banner

21 - 40 of 59 Posts

·
Registered
Joined
·
1,270 Posts
Discussion Starter #21 (Edited)
a list of pids on service 21 along with their scalars would be great.
Alrighty then let's continue. By the way I am doing this in my Rav4 which I have dubbed XV-1 which stands for Experimental Vehicle Number One. I also have a Camry dubbed XV-2. More on this one later. These are stock Toyota vehicles with stock Toyota ECU's.

Let's take a look at the Knock Feedback Value. Something initially very disheartening when going through the Canbus log file is that every single one of the Query/Response pairs for E1 (which is the presumed Knock Feedback PID) returns 0. [Correction added later: E1 is not related to knock]. For example:

02 21 E1 00 00 00 00 00
04 61 E1 00 00 00 00 00

The response is zero in this pair and it is zero in every other pair. How can this be? We know that Techstream is returning non-zero values for the Knock Feedback, so where in the heck is it getting them from. Marc you probably can guess the answer.

[Added Later: CORRECTION. It turns out that E1 has nothing to do with knock]

The answer is that the ECU packs the two knock parameters into a single response for the PID code B2. [correction was made here]. Previously, I alluded to the two "unused" parameters C and D in the standard OBD2 response,

06 61 B2 A B C D

In the previous post I've laid out how A and B are used to calculate the Knock Learn. So now I am going to demonstrate that C and D are in fact the factors for the Knock Feedback.

Again I match up the Canbus Log with the Techstream log. Looking specifically at the "MORE TIMING NEEDED EVENT," which is represented by the following time series corresponding to the jagged blue line in the previous chart under acceleration:

06 61 B2 0A 97 07 D0 00 ↔ -1.5
06 61 B2 0A 97 07 D7 00 ↔ -1.3
06 61 B2 0A 97 07 D7 00 ↔ -1.3
06 61 B2 0A 97 07 DE 00 ↔ -1.1
06 61 B2 0A 97 07 DE 00 ↔ -1.1
06 61 B2 0A 97 07 DE 00 ↔ -1.1
06 61 B2 0A 97 07 E5 00 ↔ -0.9
06 61 B2 0A 97 07 E5 00 ↔ -0.9
06 61 B2 0A 97 07 EC 00 ↔ -0.7
06 61 B2 0A A2 07 E1 00 ↔ -1
06 61 B2 0A A2 07 E1 00 ↔ -1
06 61 B2 0A A2 07 E1 00 ↔ -1
06 61 B2 0A A2 07 D0 00 ↔ -1.5
06 61 B2 0A A2 07 D0 00 ↔ -1.5
06 61 B2 0A A2 07 D0 00 ↔ -1.5
06 61 B2 0A A2 07 C8 00 ↔ -1.8
06 61 B2 0A A2 07 C1 00 ↔ -2
06 61 B2 0A A2 07 C8 00 ↔ -1.8
06 61 B2 0A A2 07 C1 00 ↔ -2
06 61 B2 0A A2 07 B2 00 ↔ -2.5
06 61 B2 0A A2 07 B9 00 ↔ -2.3
06 61 B2 0A A2 07 AA 00 ↔ -2.7
06 61 B2 0A A2 07 9B 00 ↔ -3.2
06 61 B2 0A A2 07 D0 00 ↔ -1.5

Fairly obvious, no? So to make this easier to understand, let's strip out C and D, and convert them to decimal:

(7;208) ↔ -1.5
(7;215) ↔ -1.3
(7;215) ↔ -1.3
(7;222) ↔ -1.1
(7;222) ↔ -1.1
(7;222) ↔ -1.1
(7;229) ↔ -0.9
(7;229) ↔ -0.9
(7;236) ↔ -0.7
(7;225) ↔ -1
(7;225) ↔ -1
(7;225) ↔ -1
(7;208) ↔ -1.5
(7;208) ↔ -1.5
(7;208) ↔ -1.5
(7;200) ↔ -1.8
(7;193) ↔ -2
(7;200) ↔ -1.8
(7;193) ↔ -2
(7;178) ↔ -2.5
(7;185) ↔ -2.3
(7;170) ↔ -2.7
(7;155) ↔ -3.2
(7;208) ↔ -1.5

The correlation is very clear but the one little issue we have here (that I didn't mention before) is that Techstream does not provide sufficient precision for its numbers and I had to wrestle with this a little bit for the previous PID. Or maybe it does but I don't know how to get it.

Regardless once again we posit an equation of the form

PID = x * C + y * D + z

And we have more than enough data points to solve for x, y, and z. So I'd like to see a simple way of solving this, it would help me going forward with the rest of these PID's.

And with this one solved we have the equation for PID E1 which is Knock Feedback Value. Voila.

And this is very very sneaky on the part of Toyota to pack multiple parameters into a single response. But you ain't seen nothing yet. You just wait until we get to vvti.

And by the way, I also alluded previously to the identifier 06 as opposed to 04 for the response. I believe that this identifier flags a multiple-parameter packed response.
 

·
Premium Member
Joined
·
3,511 Posts
Packing multiple pids in a single reply is a very common technique and allows dramatically increased datarates. I'm very happy to hear they are doing this.

As for the conversion, don't treat the bytes as individual, assemble the bytes first into a larger word and then you just need to figure out the scaling factor. Then realize that scaling factors are very rarely exponential, it's usually just a multiplier and an offset. at most sometimes it's an inverted value (1/x kind of thing).

For example, let's take this fictional dataset:
03, A2 : 0.82
03, EA : 0.96
01, 4C : -0.35
00, F3 : -0.53
00, 36 : -0.89

What we have going on here is a 16bit value, some multiplier, some offset and some display rounding. But knowing that they generally use powers of 2 of powers of 10, let's see if we can reverse the formula.

First let's convert the hex to numbers to make this much easier to read (note we just treated the numbers as 16 bit numbers, none of this A & B BS)
930 : 0.82
1002 : 0.96
332 : -0.35
243 : -0.53
54 : -0.89

now let's try to get the scale, so let's just subtract the smallest number on the left from all the left numbers and the same on the right (subtracting a negative so it looks like adding)
876 : 1.71
948 : 1.85
278 : 0.54
189 : 0.36
0 : 0

then let's just get the scale, so take the number on the left divided by the number on the right:
876 : 1.71 = 512.2807018
948 : 1.85 = 512.4324324
278 : 0.54 = 514.8148148
189 : 0.36 = 524.0
0 : 0

so the number is probably 512 since it's a power of two, let's make that assumption and go back the other way so we will divide the left number by our guess and see how accurate it is:
1.71 : 1.7109375
1.85 : 1.8515625
0.54 : 0.54296875
0.36 : 0.369140625

So we can see, it looks like the tool is just truncating things, 512 works as the scalar.

Great, let's work on the offset next. So let's start back with our original data set:
930 : 0.82
1002 : 0.96
332 : -0.35
243 : -0.53
54 : -0.89

and multiply our values by our new scalar:
930 : 0.82 : 419.84
1002 : 0.96 : 491.52
332 : -0.35 : -179.2
243 : -0.53 : -271.36
54 : -0.89 : -455.68

then subtract that answer (rightmost) from our original raw value:
930 : 419.84 : 510.16
1002 : 491.52 : 510.48
332 : -179.2 : 511.2
243 : -271.36 : 514.36
54 : -455.68 : 509.68

and again, we just look at the nearest power of 10 or power of two to the number on the right, again the value is 512 in this case.

So we tentatively have our formula:
(Raw-512)/512

and let's do the conversion ourselves one more time just to see where things are and how close we got:
930 : 0.82 = 0.81640625
1002 : 0.96 = 0.95703125
332 : -0.35 = -0.3515625
243 : -0.53 = -0.525390625
54 : -0.89 = -0.89453125

and we can see, YAY, everything rounds to the displayed value so we can feel confident we found the answer. Note that sometimes they'll truncate instead of round.

Does that make any sense?

But honestly if you just get me the raw bits of data with the display values i can very quickly find the formulas for you.
 

·
Registered
Joined
·
1,270 Posts
Discussion Starter #24
Quick answer for a previous question:

This is a raw CSV Export File from Techstream, including time stamps.

A few little notes:

1. I've changed the file type from .csv to .txt because .csv is not allowed as an attachment here.

2. Nobody told the persons at Toyota that .csv stands for "Comma Separated Values." So even though they give their export filetype as .csv, they use the tab character as the data delimiter. Ha ha. Ha ha ha ha ha.

3. With my Windows 10 version of Techstream I'm using a newer J2534 chipset cable (Mongoose clone) with a faster data rate than older knockoffs... judging from the export file, the overall data rate seems to be about 1/0.07 or about 15 PID/sec in Techstream, which is an improvement over what I had previously with an older cable.

4. Even in the export file, the precision of the data leaves something to be desired. It's fixed precision one decimal point, same as the software's data display. Most likely the persons at Toyota did this with a character data format in mind, not a numeric precision. IDK.

One of the luxuries and great pleasurable joys of reverse engineering is you get to second-guess the decisions of the original development team. Ha ha. Ha ha ha ha ha.
 

Attachments

·
Premium Member
Joined
·
3,511 Posts
Yeah, I've spent many years and an engineer and i am always amazed at how often i run across other engineers that have no damn concept of significant figures. It's annoying every time.

If you want to work on this together I'd be happy to analyze the csv files you post and we can make the magic decoder ring for the 2GR ECU. Once we get this working I might see about making a fast datalogger for this thing. I'll hammer it directly with a microcontroller to see if we can get high data rates.

If you are interested, can we start with a list of all the parameter names, units and default value at idle?

Thanks!
 

·
Registered
Joined
·
1,270 Posts
Discussion Starter #26
Ok I'm willing to do that. I'll start putting things together. Basically I'll continue to do what I'm doing and I'll share the Techstream and CANbus logfiles with you. The engine control stuff is organized into groupings that I'll tackle one by one or maybe to keep things manageable I might make subgroupings like the one I just did for knock. But you should be aware that there is some stuff that seems pretty seriously obfuscated and I don't know if this was on purpose or just a by-product of Mr. Tee's general policy for data handling that we already know and love from messing with the ECU files. Surprise, huh?

It seems to me that some other developers are including PID logging capability in their so-called tuning software suites and I know they keep this stuff tightly under wraps. I would like to keep this an open and free resource as much as possible for everyone to benefit.
 

·
Premium Member
Joined
·
3,511 Posts
That's awesome and don't stress the obfuscation, I'm pretty sure i can deal with all of that. The tricky part will be that some of these PIDs will be hard to get to change in a meaningful way so that we can eliminate enough rounding error to get the right scaling factors. For some of these I may have to make special tunes to do the testing with. But the big thing i need your help with is associating PID IDs with PID names.

Are you ok just doing this as a shared google document? we can get edit permissions for ourselves and anyone else that can reliably help but make it viewable to the world?
 

·
Registered
Joined
·
1,270 Posts
Discussion Starter #28 (Edited)
Ok I'll be getting the PIDs and PID names listed. This weekend I should have some time to get a few.

Let's wrap up the loose end of the equation for Knock Feedback which is calculated from variables C and D in the knock response string. Going back it's apparent from the list that the variable C is invariant so it plays no part. This leaves a factor for D and an offset to be determined. Fairly simple to eyeball and determine that the equation is:

knock feedback = (D - 256)/32

And here is the verification:

208 : -1.5 ↔ -1.50
215 : -1.3 ↔ -1.28
215 : -1.3 ↔ -1.28
222 : -1.1 ↔ -1.06
222 : -1.1 ↔ -1.06
222 : -1.1 ↔ -1.06
229 : -0.9 ↔ -0.84
229 : -0.9 ↔ -0.84
236 : -0.7 ↔ -0.63
225 : -1 ↔ -0.97
225 : -1 ↔ -0.97
225 : -1 ↔ -0.97
208 : -1.5 ↔ -1.50
208 : -1.5 ↔ -1.50
208 : -1.5 ↔ -1.50
200 : -1.8 ↔ -1.75
193 : -2 ↔ -1.97
200 : -1.8 ↔ -1.75
193 : -2 ↔ -1.97
178 : -2.5 ↔ -2.44
185 : -2.3 ↔ -2.22
170 : -2.7 ↔ -2.69
155 : -3.2 ↔ -3.16
208 : -1.5 ↔ -1.50

There's one more little loose end. If anyone has been paying attention, because my log files show requests for both E1 and B2, and the return values are packed into a single response, I am not able to determine which is which. This means that the equation for B2 could be the equation for E1 and vice versa. So I need to re-do two quick logs (or at least one) to request E1 and B2 individually, and then this ambiguity will be resolved.

PS. Correction: It turns out that E1 has nothing to do with knock. It is a frame delimiter. The knock learn and knock feedback share the same PID code, B2.
 

·
Premium Member
Joined
·
12,678 Posts
There's one more little loose end. If anyone has been paying attention, because my log files show requests for both E1 and B2, and the return values are packed into a single response, I am not able to determine which is which. This means that the equation for B2 could be the equation for E1 and vice versa. So I need to re-do two quick logs (or at least one) to request E1 and B2 individually, and then this ambiguity will be resolved.
Didn't you solve all of that by knowing to begin with that Knock Feedback was a value in the -1.5ish range, and Knock Learn was a value in the 20ish range?
 

·
Registered
Joined
·
1,270 Posts
Discussion Starter #30
Didn't you solve all of that by knowing to begin with that Knock Feedback was a value in the -1.5ish range, and Knock Learn was a value in the 20ish range?
That "feels" right but I have no a priori knowledge of what these parameters signify or what their expected values are and to my knowledge they have not been discussed anywhere in the Toyota publicverse so I want to make absolutely certain that I'm getting it right. Disambiguation.
 

·
Premium Member
Joined
·
12,678 Posts
That "feels" right but I have no a priori knowledge of what these parameters signify or what their expected values are and to my knowledge they have not been discussed anywhere in the Toyota publicverse so I want to make absolutely certain that I'm getting it right. Disambiguation.
I thought those values and their association to the parameters came from your Techstream log? Perhaps I am misunderstanding what you get from Techstream?
 

·
Registered
Joined
·
1,270 Posts
Discussion Starter #32 (Edited)
I thought those values and their association to the parameters came from your Techstream log? Perhaps I am misunderstanding what you get from Techstream?
Techstream does not identify the PID code. It just gives the PID name i.e Knock Feedback Value or Knock Learn Value. The CANbus log does not identify the PID name. It just gives the PID code, ie. B2 or E1. So if there are two parameters being logged, and the response for the two is combined, it's not possible to know which PID code corresponds to which PID name, without making a guess which is an unsupported inference. I am just taking one more step to validate the guess. If the Canbus log includes only one PID code request and the Techstream log includes only one PID name then we know that the PID code corresponds to the PID name unambiguously.

PS. The justification for this step will become more clear when we deal with an array-type response that packs eight or 10 parameters. Coming soon.
 

·
Premium Member
Joined
·
3,511 Posts
For the PIDs that are harder to identify and scale it could be worth just injecting your own PID answers on the CAN bus and see what the TIS system translates them to. Then you can test the full range and get a much better idea of the scale and offsets.
 

·
Registered
Joined
·
1,270 Posts
Discussion Starter #34
NEWSFLASH: Talk about disambiguation E1 is not even a knock related PID code. It is some kind of secret handshake code between Techstream and the ECU. It shows up in the log for knock because... because. And now we really don't know whether B2 refers to knock feedback or knock learn - or both. But we're gonna figure it out.
 

·
Registered
Joined
·
1,270 Posts
Discussion Starter #35 (Edited)
Q. How did I reach the conclusion that the PID code E1 is not related to knock?

A. Simple, I started inspecting CANbus log files, and E1 query and response frames appeared in many of them, without being associated with any specific parameters.

1. CANbus log for knock - E1 query and response circled in blue.



2. CANbus log for Transmission fluid temperature - E1 query and response circled in red.



3. CANbus log for VVT - E1 query and response circled in red.



4. CANbus log for fuel injection time and volume - E1 query and response circled in red.



I want to emphasize that these E1 queries are being performed by Techstream without in any way being requested by the user (me). So E1 is like the movie character Zelig, inserting itself into every frame, without contributing anything. Ha ha.

[Added for clarification: It does look like E1 is a frame delimiter of some kind. It seems to indicate that all the parameters in one frame have been queried and responded and a new frame can begin. It does occur at the beginning of every frame.]

Q. So where does this leave us with respect to the knock PID code?

A. It is apparent that the two parameters knock feedback and knock learn share a single PID code, B2, and the scaling factors for both parameters are embedded in the response for the code B2.

So with this out of the way we are ready to move on to more even better stuff. Believe you me there is a lot more even better stuff coming. Ha ha.
 

·
Premium Member
Joined
·
3,511 Posts
That E1 is likely a heartbeat used by techstream to make sure it still had a good connection to the ECU or something like that. It might also be some ignition state that shows up in the toolbar instead of in your requested PID list.
 

·
Registered
Joined
·
1,270 Posts
Discussion Starter #37
Here's a partial list of the parameters that Techstream has in it for the 2008 2GR-FE Rav4 4wd. It's partial only because I cannot get Techstream to spit out out the entire list all at once. I'll revise this later with the missing parameters. Right here we have 119 parameters. Some of these parameters share PIDs, others are in the public OBD2 standard, so I haven't pinpointed the exact number of PID's to study yet. These are the parameter names and, in parentheses, the units, exactly as printed in the Techstream CSV log.

Sample Time (MM:SS.mmm)
Vehicle Speed (km/h)
Engine Speed (rpm)
Calculate Load (%)
Vehicle Load (%)
MAF (gm/s)
Atmosphere Pressure (kPa)
Coolant Temp (?)
Intake Air (?)
Engine Run Time (s)
Initial Engine Coolant Temp (?)
Initial Intake Air Temp (?)
Battery Voltage (V)
Accel Sens. No.1 Volt % (%)
Accel Sens. No.2 Volt % (%)
Throttle Sensor Volt % (%)
Throttl Sensor #2 Volt % (%)
Throttle Idle Position ()
Throttle Require Position (V)
Throttle Sensor Position (%)
Throttle Position No.1 (V)
Throttle Position No.2 (V)
Throttle Position Command (V)
Throttle Sens Open Pos #1 (V)
Throttle Sens Open Pos #2 (V)
Throttle Motor Current (A)
Throttle Motor DUTY (%)
Throttle Motor Duty (Open) (%)
Throttle Motor Duty (Close) (%)
Throttle Fully Close Learn (V)
Injector (Port) (ms)
Injection Volum (Cylinder1) (ml)
Fuel Pump Speed Control ()
Fuel Pump/Speed Status ()
Vacuum Pump ()
EVAP (Purge) VSV (%)
Evap Purge Flow (%)
Purge Density Learn Value ()
Vapor Pressure Pump (kPa)
Vapor Pressure (Calculated) (kPa)
EVAP System Vent Valve ()
EVAP Purge VSV ()
Target Air-Fuel Ratio ()
AF Lambda B1S1 ()
AF Lambda B2S1 ()
AFS Voltage B1S1 (V)
AFS Voltage B2S1 (V)
AFS Current B1S1 (mA)
AFS Current B2S1 (mA)
O2S B1S2 (V)
O2S B2S2 (V)
O2S Impedance B1S2 (ohm)
O2S Impedance B2S2 (ohm)
Short FT B1S1 (%)
Long FT B1S1 (%)
Total FT #1 ()
Short FT B2S1 (%)
Long FT B2S1 (%)
Total FT #2 ()
Fuel System Status #1 ()
Fuel System Status #2 ()
IGN Advance (deg)
Knock Feedback Value (deg(CA))
Knock Correct Learn Value (deg(CA))
ACIS VSV ()
AICV VSV ()
Catalyst Temp B1S1 (?)
Catalyst Temp B2S1 (?)
Catalyst Temp B1S2 (?)
Catalyst Temp B2S2 (?)
Starter Signal ()
Power Steering Signal ()
Power Steer. Sig. Record ()
Neutral Position SW Signal ()
Stop Light Switch ()
A/C Signal ()
Closed Throttle Position SW ()
Electrical Load Signal ()
# Codes(Include History) ()
MIL ON Run Distance (Km)
Running Time from MIL ON (min)
Time after DTC Cleared (min)
Distance from DTC Cleared (km)
Warmup Cycle Cleared DTC ()
TC and TE1 ()
Ignition Trig. Count ()
Cylinder #1 Misfire Count ()
Cylinder #2 Misfire Count ()
Cylinder #3 Misfire Count ()
Cylinder #4 Misfire Count ()
Cylinder #5 Misfire Count ()
Cylinder #6 Misfire Count ()
All Cylinders Misfire Count ()
Misfire RPM (rpm)
Misfire Load (g/rev)
Misfire Margin (%)
Battery Current (A)
Battery Temperature (?)
Alternator Output Duty (%)
Alt Vol - Non Active Test (V)
Alt Vol - Active Test (V)
Electric Fan Motor ()
Idle Fuel Cut ()
FC TAU ()
SPD (NT) (rpm)
SPD (NC) (rpm)
Shift SW Status (P Range) ()
Shift SW Status (R Range) ()
Shift SW Status (2 Range) ()
Shift SW Status (L Range) ()
Shift SW Status (N Range) ()
Shift SW Status (D Range) ()
Shift SW Status (4 or D) ()
Shift SW Status (3 Range) ()
A/T Oil Temperature 1 (?)
Lock Up ()
Lock Up Solenoid Status ()
Shift Status ()
SLT Solenoid Status ()
 

·
Premium Member
Joined
·
3,511 Posts
That's a good list, there's a few juicy things in there. If i could guide your efforts i'd love to put these higher in the priority list:
Calculate Load (%)
Vehicle Load (%)
MAF (gm/s)
Injector (Port) (ms)
Injection Volum (Cylinder1) (ml)
AF Lambda B1S1 ()
AF Lambda B2S1 ()
Throttle Idle Position ()
Throttle Require Position (V)
Throttle Sensor Position (%)
Target Air-Fuel Ratio ()
Engine Speed (rpm)
IGN Advance (deg)
Knock Feedback Value (deg(CA))
Knock Correct Learn Value (deg(CA))
Short FT B1S1 (%)
Long FT B1S1 (%)
Total FT #1 ()
Short FT B2S1 (%)
Long FT B2S1 (%)
Total FT #2 ()

But in the end, even the ones that are duplicates of normal mode 02 OBD requests would be good to investigate because if this mode has them packed in a single reply we can get those for free.
 

·
Registered
Joined
·
1,270 Posts
Discussion Starter #39
Here's some very juicy fueling results:

Name = Injection Volum (Cylinder1) (ml)
PID = F3
QUERY = 02 21 F3 00 00 00 00 00
RESPONSE = 07 61 F3 A B C D E
EQUATION = =(8*A + B/32)/1000

Name = Injector (Port) (ms)
PID = F3
QUERY = 02 21 F3 00 00 00 00 00
RESPONSE = 07 61 F3 A B C D E
EQUATION = 128*C/1000

There is not enough variation in D and E to determine any correlation with the precision provided in the dependent variables.

Note the two variables are strongly correlated but not exactly the same. Here's how they look together on a chart:



This is an idle rev followed by a partial throttle changing to full throttle.

Log files attached.

I didn't log load but just guesstimating what it was it looks like Injection Volume correlates with enrichment factor * this map



divided by 256.
 

Attachments

·
Premium Member
Joined
·
3,511 Posts
Port injection time and injection volume differ a bit because of opening time delays and manifold pressure compensation, though if it does the manifold pressure compensation this is an estimate since it has no manifold pressure sensor. The ECU does have a manifold pressure compensation table though. But, honestly i would not trust the volume, it's just an estimate and the ECU has so many little lies on so many maps. But, with the injection time we can now get a duty cycle without probing wires!

Injection time/(1.2/RPM) = duty cycle percentage

So that right there is a huge win already.
 
21 - 40 of 59 Posts
Top