The VBA "call-stack" and calling mechanisms
Using VarPtr() and some memory dump in a VBA Standard Module reveals the following. VBA run-time allocation of local variables, passing parameters, temp-variables, function calls uses a common memory area, which looks very similar to stack-frame structures used by other compilers, like C.
This VBA "stack" is certainly not the CPU stack - only that the memory layout and the operation looks very similar:
- caller "pushes" parameters onto stack, then execution transferred to callee (call)
- callee "pulls" the stack pointer to allocate space for local variables
- when function code execution ends, both callee and the caller "cleans up" and restores the stack.
There is a little difference between 64- and 32-bit VBA run-time when it comes to memory layout:
- 64-bit run-time
We find the parameter- and local block together in memory (similar to a traditional stack).
- 32-bit run-time
The two blocks are separated in memory. Caller builds up the complete parameter block locally and the function receives a pointer to the parameter block together with locals.
The final operation will be the same for both, only parameter access differs. Here "push" means copy a value into the parameter block by the caller; the CALL-line tries to denote where execution transfers to another function:
increasing
memory address
^
|
| 64-bit VBA "call-stack" 32-bit VBA "call-stack"
|
| | | | |
| | local1 | | local1 |
| | local2 | | local2 |
| | local4 | Caller | local4 | Caller
| |_________________| local block |___________| local block
| | ... | | ... |
| | ... | | ... |
| | ... | | push |
| | ... | | | |
| | ... | |_____V_____|
| | push | | p3 | callee
| | | | | p2 | parameter block
| |________V________| --> | p1 |
| | p3 | callee | |___________|
| | p2 | parameter block | | ... |
| | p1 | | ...
| __|_________________|________________ CALL | ...
| | local1 | callee | |___________|________________ CALL
| | local2 | local block --- | [ptr] |
| |_________________| | local1 | callee
| | ... | | local2 | local block
| | ... | |___________|
| | ... | | ... |
| | | | |
| | | | | | |
| | |
| V V
STACK STACK
VBA Function: return value, locals and parameters memory layout
This was made with vbadump and in 64-bit VBA.
Follows a VBA function in a Standard Module. It has local variables, returns LongPtr and has 2 ByVal arguments (so we can look at):
Private Function func(ByRef p1 As LongPtr, ByRef p2 As LongPtr) As LongPtr Dim L As Long Dim I As Integer Dim p As LongPtr p = -1 L = -1 I = -1 func = 9 memdump VarPtr(p), 10 ' peek End Function
The "stack" memory layout:
Lets call this function with (1, 2) and dump locals starting from the memory address of p:
"stack"
memory
address
^
| 00000234EBCB29B8: 0000000000000000 ..
| 00000234EBCB29B0: 0200000000000000 p2
| 00000234EBCB29A8: 0100000000000000 p1
| 00000234EBCB29A0: F08EC7EB34020000
| 00000234EBCB2998: 0900000000000000 func
| 00000234EBCB2990: FFFFFFFF00000000 L
| 00000234EBCB2988: FFFF000000000000 I
| 00000234EBCB2980: FFFFFFFFFFFFFFFF p <--- VarPtr(p)
|
In 64-bit VBA all variables are naturally 8-byte aligned. The structure indeed looks like a stack-frame of a C compiler: locals on lower addresses, parameters higher.
Function "return value"
The Function "return value" is a hidden local variable, as if it was declared as the very first local variable in the function:
Private Function func(ByRef p1 As LongPtr, ByRef p2 As LongPtr) As LongPtr [Dim func As LongPtr] Dim L As Long Dim I As Integer
It makes sense, the syntax is also the same: we use "func" as if it were a normal variable, like func=100, and on exiting the function the run-time takes care of it to return its value somehow to the caller.
With this method we can see the memory map for other, more "interesting" data types by inserting a Dim statement between L and I:
00000234EBCB29B0: 0200000000000000 p2 00000234EBCB29A8: 0100000000000000 p1 00000234EBCB29A0: F08EC7EB34020000 00000234EBCB2998: 0900000000000000 func 00000234EBCB2990: FFFFFFFF00000000 Long 00000234EBCB2988: 0000000000000000 String <-- null-pointer 00000234EBCB2980: FFFF000000000000 Integer 00000234EBCB2978: FFFFFFFFFFFFFFFF LongPtr 000001E8925DC050: 0200000000000000 p2 000001E8925DC048: 0100000000000000 p1 000001E8925DC040: 28D0689AE8010000 000001E8925DC038: 0900000000000000 func 000001E8925DC030: FFFFFFFF00000000 Long 000001E8925DC028: 0000000000000000 -- 000001E8925DC020: 6400000000000000 | 64-bit Variant = 100 (vbInteger=2) 000001E8925DC018: 0200000000000000 -- 000001E8925DC010: FFFF000000000000 Integer 000001E8925DC008: FFFFFFFFFFFFFFFF LongPtr 00000234EBCB29B0: 0200000000000000 p2 00000234EBCB29A8: 0100000000000000 p1 00000234EBCB29A0: F08EC7EB34020000 00000234EBCB2998: 0900000000000000 func 00000234EBCB2990: FFFFFFFF00000000 Long 00000234EBCB2988: 0000000000000000 Object <-- null-pointer 00000234EBCB2980: FFFF000000000000 Integer 00000234EBCB2978: FFFFFFFFFFFFFFFF LongPtr 00000234EBCB29B0: 0200000000000000 p2 00000234EBCB29A8: 0100000000000000 p1 00000234EBCB29A0: F08EC7EB34020000 00000234EBCB2998: 0900000000000000 func 00000234EBCB2990: FFFFFFFF00000000 Long 00000234EBCB2988: 0000000000000000 varr() <--- Dynamic Array, null-pointer 00000234EBCB2980: FFFF000000000000 Integer 00000234EBCB2978: FFFFFFFFFFFFFFFF LongPtr 00000234EBCB29B0: 0200000000000000 p2 00000234EBCB29A8: 0100000000000000 p1 00000234EBCB29A0: F08EC7EB34020000 00000234EBCB2998: 0900000000000000 func 00000234EBCB2990: FFFFFFFF00000000 Long 00000234EBCB2988: 0400000000000000 -- 00000234EBCB2980: 70586C8134020000 | 00000234EBCB2978: 0000000000000000 | varr(3) <--- Fixed-size array: 64-bit SAFEARRAY 00000234EBCB2970: 0100920818000000 | STRUCTURE 00000234EBCB2968: 000000000C000000 -- 00000234EBCB2960: FFFF000000000000 Integer 00000234EBCB2958: FFFFFFFFFFFFFFFF LongPtr 00000234EBCB29B0: 0200000000000000 p2 00000234EBCB29A8: 0100000000000000 p1 00000234EBCB29A0: F08EC7EB34020000 00000234EBCB2998: 0900000000000000 func 00000234EBCB2990: FFFFFFFF00000000 Long 00000234EBCB2988: 3100320033003400 String * 4 = "1234" <--- 2x4=8 bytes of unicode fix-string 00000234EBCB2980: FFFF000000000000 Integer 00000234EBCB2978: FFFFFFFFFFFFFFFF LongPtr
VBA calling mechanisms and memory map
Here we look at how parameters are passed in memory between Sub/Function-s inside a Standard VBA Module. What exactly ByRef/ByVal means for different VBA data types, and what happens when a Sub/Function expects a Variant argument: how the caller performs the implicite conversion.
This was done dumping the "stack" way above the current Sub/Function to reveal the caller's local area as well.
For visualizing the so called "call-stack" of the running VBA code:
- each box is an allocated data structure or local variable as it appears in memory dump
- the caller function's local variables, temp variables are on the top, the stack "grows", then execution transferred to another function: that is the "call-line" --------
- on entering a function the "stack-pointer" is at the last parameter "pushed". The function allocates further stack space for it's local variables etc.
These are the main data type groups we look at each:
- scalars (1-8 bytes): Integer, Long, LongLong, Byte, Single, Double, Boolean, Currency, Date
- Variant - 16/24-byte structure 32/64-bit VBA
- String - is a pointer
- Object - is a pointer
- Dynamic Array - is a pointer
- Fixed-size Array - is a structure
How VBA stores its data types is essential here to understand first - see it in another chapter.
The caller has some different types of local varaiables, f. ex.:
Private Sub caller() Dim L As Long Dim varr(3) Dim I As Integer Dim v Dim s As String Dim coll As Collection Dim B As Byte Dim CY As Currency Dim p As LongPtr ' calls different sub and pass values End Sub
Passing matchig data types ByRef/ByVal
As a general rule:
ByRef:
- caller passes a pointer to variable or a result of an expression stored on stack.
- callee expects a pointer of a specific type. Compile time check.
- callee dereferences the pointer quietly each time of access.
- valid for all data types
ByVal:
- push makes a copy of the variable or expression result
- valid for all data types except Array
Quick comparison:
Pass ByRef Pass ByVal
_______ _______
| I | Dim I As Integer | I | Dim I As Integer
|_______| <--- <-- VarPtr(i) |_______|
|
| push | | push
V address of | V
______________ | _______
| pointer | i | | copy | i <-- VarPtr(i)
|______________| |_______|
---------------------------- Sub (ByRef i As Integer) -------------------------- Sub (ByVal i As Integer)
ByRef: VarPtr(i) gives the address of the variable of the caller - i.e. the value stored in the passed pointer.
ByVal: VarPtr(i) gives the address of the pushed copy.
When a ByRef parameter is passed to another Sub again ByRef, the pointer itself is pushed to Sub2() so that Sub2() can work again on the original variable of the caller. Note VarPtr(i2):
Pass ByRef to Sub2:
_______
| I | Dim I As Integer
|_______| <--- <-- VarPtr(i) points here
| <-- VarPtr(i2) also points here
| push |
V address of |
______________ |
| pointer | i |
|______________|
---------------------------- Sub (ByRef i As Integer)
|
|
|
| push |
V |
______________ |
| pointer | i2 |
|______________|
---------------------------- Sub2 (ByRef i2 As Integer)
Passing Arrays
There are two types of array declaration in VBA:
- Dynamic Array: Dim varr()
- Fixed-size Array: Dim arr(3)
A Dynamic Array variable is a pointer to a COM SAFEARRAY structure (PSA). A Fixed-size Array is-a COM SAFEARRAY structure. For Dim that means that a Dynamic Array variable allocates pointer-size (4/8 bytes), while Fixed-size Array allocates the whole SAFEARRAY structure in local memory.
Passing array is:
- only ByRef
- callee expects an address of a Pointer to SAFEARRAY (PSA)
- requires temp PSA by caller for Fixed-size Arrays
- Dynamic Array variable is-a PSA (the mechanism will be the same as with general ByRef)
- VarPtr() is N/A for Arrays
Pass Fixed-size Array:
______________
| | Dim arr(3)
| SAFEARRAY |
| STRUCT |
| | <---
|______________| |
|
| temp ptr |
V | Pass Dynamic Array:
______________ | ______________
| PSA -> | | | PSA -> | Dim varr()
|______________| <--- |______________| <---
| |
| push | | push |
V address of | V address of |
______________ | ______________ |
| pointer | | | pointer | |
|______________| |______________|
------------------------------- Sub ([ByRef] varr()) -------------------------- Sub ([ByRef] varr())
Passing String
Pass String ByVal: Pass String ByRef:
______________ ______________
| --> BSTR1 | Dim S As String | --> BSTR | Dim S As String
|______________| |______________| <---
|
| push | push |
V V address of |
______________ ______________ |
| --> BSTR1 | | pointer | |
|______________| |______________|
------------------------- Sub (ByVal s As String) ------------------------- Sub (ByRef s As String)
| deep copy
V
______________
| --> BSTR2 | <-- VarPtr(s) points here
|______________| hidden local var
Passing String ByRef is the same mechanism as general ByRef.
Passing String ByVal:
Caller makes exact copy of the String variable for push.
It is callee that performs the actual deep-copy, allocates new BSTR and a copy of the original string bytes itself.
The resulting String variable is stored locally as if it was declared here - and VarPtr(i) gives the address of this "hidden" variable:
Sub (ByVal s As String) [Dim s As String] Dim ... .. End Sub
Passing Variant ByVal
- Caller pushes an exact copy of the Variant structure and makes the call.
- Callee always allocates a new, hidden Variant locally and copies the Variant parameter. This is because at run-time the exact data type stored in the Variant is not known: callee will perform simple- or deep-copy based on the received VbVarType of the Variant.
Passing Variant of scalars ByVal
Pass Variant/Integer ByVal: simple copy for scalars
____ _________ ______________
|02 | | 100 | Dim V
|____|_________|______________| V = 100
| push
V
____ _________ ______________
|02 | | 100 |
|____|_________|______________|
----------------------------------------- Sub (ByVal v)
| copy
V
____ _________ ______________
|02 | | 100 | <-- VarPtr(v) points here
|____|_________|______________| hidden local var
02 denotes vbVarType = vbInteger.
Note. This is also true for expressions results, f. ex. sub_v_ByVal 100: first a new Variant holding the result is created on stack, then this value is copied and pushed.
Passing Variant/String and Variant/Array ByVal
Passing Strings and Arrays ByVal stored in Variants uses the same mechanism but with deep-copy: on entering the function a new BSTR or a new SAFEARRAY structure will be allocated and content of the data it's holding copied.
Pass Variant/String ByVal Pass Variant/Array ByVal
____ _________ ______________ ____ _________ ______________
|08 | | --> BSTR1 | Dim V |VT_ARRAY | --> SA1 | Dim V
|____|_________|______________| V = "s" |____|_________|______________| ReDim V(3)
| push | push
V V
____ _________ ______________ ____ _________ ______________
|08 | | --> BSTR1 | |VT_ARRAY | --> SA1 |
|____|_________|______________| |____|_________|______________|
-------------------------------------- Sub (ByVal v) -------------------------------------- Sub (ByVal v)
| deep copy | deep copy
V V
____ _________ ______________ ____ _________ ______________
|08 | | --> BSTR2 | <-- VarPtr(v) points here |VT_ARRAY | --> SA2 | <-- VarPtr(v) points here
|____|_________|______________| hidden local var |____|_________|______________| hidden local var
08 denotes vbVarType = vbString. The VT_ARRAY flag is a VARENUM enumeration declared in wtypes.h of the Automation API (COM) - same as vbArray.
Passing other data types and expressions As Variant
This is the case when a Sub/Function is declared with a Variant argument:
Sub|Function (ByRef|ByVal v [As Variant])
Anything, any variable or expression can be passed to such a function ByRef or ByVal. This is possible because Variant can store any other data type and the conversion is implicite in VBA.
Callee still expects Variant - it is the caller that makes the necessary conversions.
Passing other data types As ByRef Variant
- Caller creates a special temp Variant with VT_BYREF flag set and places a pointer into the Variant.
VbVarType - including the VT_ARRAY flag - is copied/set correctly for callee. - Callee recognises this VT_BYREF Variant and quietly dereferences the pointer. So this is a double dereference each time of access.
The mechanism is essentially the same for all dataypes:
- scalars
- String
- Dynamic Array (PSA)
Pass scalar As ByRef Variant:
_______
| 100 | Dim I As Integer
|_______| <---
|
| temp |
V |
____ _________ ______________
|VT_BYREF | pointer | <-- VarPtr(v) points here(!)
|____|_________|______________| <---
|
| push address of |
V |
______________ |
| pointer | v |
|______________|
----------------------------------------- Sub (ByRef v)
Note that VarPtr(v) points to the temp VT_BYREF Variant. Seems like VBA simply returns the received ByRef pointer value itself for VarPtr(v).
Also, similar to passing a ByRef parameter to another Sub again ByRef, the pointer itself is pushed. Note VarPtr(v2) also points to the original temp VT_BYREF Variant of the caller:
Pass scalar As ByRef Variant to Sub2:
_______
| 100 | Dim I As Integer
|_______| <---
|
| temp |
V |
____ _________ ______________
|VT_BYREF | pointer | <-- VarPtr(v) points here
|____|_________|______________| <--- <-- VarPtr(v2) also points here
|
| push address of |
V |
______________ |
| pointer | v |
|______________|
----------------------------------------- Sub (ByRef v)
|
|
|
| push |
V |
______________ |
| pointer | v2 |
|______________|
----------------------------------------- Sub2 (ByRef v2)
The mechanism is the same for String and Array (just for reference):
Pass String as ByRef Variant: Pass Dynamic Array As ByRef Variant:
______________ ______________
| --> BSTR | <--- Dim str as String | --> SA | <--- Dim varr()
|______________| | |______________| |
| |
| temp | | temp |
V | V |
____ _________ ______________ | ____ _________ ______________ |
|VT_BYREF | pointer | <--- |VT_BYREF | pointer | <---
|____|_________|______________| | |____|_________|______________| |
| |
| push address of | | push address of |
V | V |
______________ | ______________ |
| pointer | v | | pointer | v |
|______________| | |______________| |
----------------------------------------- Sub (ByRef v) ----------------------------------------- Sub (ByRef v)
As with passing arrays the function expects a pointer to PSA. For Fixed-size Arrays the caller additionally creates a temp PSA on stack. This is the memory layout:
Pass Fixed-size Array as ByRef Variant:
______________
| | Dim arr(3)
| SAFEARRAY |
| STRUCT |
| | <---
|______________| |
|
| temp ptr |
V |
______________ |
| PSA -> | <---
|______________| |
|
| temp Variant |
V |
____ _________ ______________ |
|VT_BYREF | pointer | <---
|____|_________|______________| |
|
| push address of |
V |
______________ |
| pointer | v |
|______________|
----------------------------------------- Sub (ByRef v)
Passing other data types As ByVal Variant
Caller allocates a temp Variant and copies the source data as-is. This "cloning" does not create any copy of the BSTR or SAFEARRAY.
Callee performs the same operation as with any ByVal Variant: allocates a new, hidden Variant locally and copies the Variant parameter, which can be simple- or deep-copy (String and Array).
Pass scalar As ByVal Variant:
_______
| 100 | Dim I As Integer
|_______|
| clone
V
____ _________ ______________
|02 | | 100 |
|____|_________|______________|
| push
V
____ _________ ______________
|02 | | 100 |
|____|_________|______________|
------------------------------------- Sub (ByVal v)
| copy
V
____ _________ ______________
|02 | | 100 | <-- VarPtr(v)
|____|_________|______________| local hidden copy
For String and Array involves deep copy (note BSTR2 and SA2):
Pass String As ByVal Variant: Pass Dynamic Array As ByVal Variant:
______________ ______________
| --> BSTR1 | Dim str as String | --> SA1 | Dim varr()
|______________| |______________|
| clone | clone
V V
____ _________ ______________ ____ _________ ______________
|08 | | --> BSTR1 | |VT_ARRAY | --> SA1 |
|____|_________|______________| |____|_________|______________|
| push | push
V V
____ _________ ______________ ____ _________ ______________
|08 | | --> BSTR1 | |VT_ARRAY | --> SA1 |
|____|_________|______________| |____|_________|______________|
------------------------------------- Sub (ByVal v) ------------------------------------- Sub (ByVal v)
| deep copy | deep copy
V V
____ _________ ______________ ____ _________ ______________
|08 | | --> BSTR2 | <-- VarPtr(v) |VT_ARRAY | --> SA2 | <-- VarPtr(v)
|____|_________|______________| local hidden copy |____|_________|______________| local hidden copy
When passing Fixed-size Array caller again creates a temp VT_ARRAY Variant holding the address of the SAFEARRAY structure:
Pass Fixed-size Array As ByVal Variant:
______________
| | Dim arr(3)
| SAFEARRAY |
| STRUCT |
| | <---
|______________| |
|
| clone |
V |
|
____ _________ ______________
|VT_ARRAY | --> SA1 |
|____|_________|______________|
| push
V
____ _________ ______________
|VT_ARRAY | --> SA1 |
|____|_________|______________|
----------------------------------------- Sub (ByVal v)
| deep copy
V
____ _________ ______________
|VT_ARRAY | --> SA2 | <-- VarPtr(v) points here
|____|_________|______________| hidden local var
Passing Object variables
Passing Objects in VBA ByRef follows the same mechanism as other pointer data types, like String.
Passing Objects ByVal is a different mechanism.
Object is the only true reference type in VBA and more than one pointer is allowed to hold the address of the same object. This is because of the IUnknown Interface and reference counting. There is no "deep copy" of an Object type i.e. we cannot make another full copy with items of a Collection f. ex. by assignment - opposed to Strings and Arrays.
This is also true when Variants are holding a reference to an Object:
Dim v1, v2 Set v1 = New JVAR Set v2 = v1 v1 v2 0900-000000000000-4090800716020000-0000000000000000 0900-000000000000-4090800716020000-0000000000000000 | | +------------> same instance <------------------------+ 09 = vbObject
There is no difference in the ByVal calling mechanism whether typed- or general Object reference is passed, or callee expects typed- or general Object:
Pass Object ByVal:
______________ ______________
| --> ref | Dim coll As Collection | --> ref | Dim coll As Collection
|______________| |______________|
| clone | clone
V V
______________ ______________
| --> ref | | --> ref |
|______________| |______________|
| push | push
V V
______________ ______________
| --> ref | | --> ref |
|______________| |______________|
------------------------ Sub (ByVal o As Object) ------------------------ Sub (ByVal coll As Collection)
| copy | copy
V V
______________ ______________
| --> ref | <-- VarPtr(o) | --> ref | <-- VarPtr(coll)
|______________| local hidden copy |______________| local hidden copy
Why the clone-ing step happens is unclear. The only "benefit" of a ByVal Object is that the Sub/Function cannot change the object reference itself (it made a copy). It can add/remove items from the Collection but cannot deallocate the original object by setting the reference to Nothing, create a new one etc.
No comments:
Post a Comment