TL;DR :
Question #1:
POPF doesn't actually allow you to change the VM flag per the Instruction Set Architecture reference:
When operating in protected, compatibility, or 64-bit mode at privilege level 0 (or in real-address mode, the equivalent to privilege level 0), all non-reserved flags in the EFLAGS register except RF1, VIP, VIF, and VM may be modified. VIP, VIF and VM remain unaffected.
There are two general mechanisms that can be used to set EFLAGS.VM and enter v8086 mode:
A task switch to an 80386 task loads the image of EFLAGS from the new TSS. The TSS of the new task must be an 80386 TSS, not an 80286 TSS, because the 80286 TSS does not store the high-order word of EFLAGS, which contains the VM flag. A value of one in the VM bit of the new EFLAGS indicates that the new task is executing 8086 instructions; therefore, while loading the segment registers from the TSS, - the processor forms base addresses as the 8086 would.
An IRET from a procedure of an 80386 task loads the image of EFLAGS from the stack. A value of one in VM in this case indicates that the procedure to which control is being returned is an 8086 procedure. The CPL at the time the IRET is executed must be zero, else the processor does not change VM.
Question #2:
v8086 mode is only available on an x86-64 processor in 32-bit protected mode (legacy mode). You can not use it in 64-bit mode or 32-bit (or 16-bit) compatibility modes. You would have to switch the processor out of long mode and enter 32-bit protected mode (legacy mode) running at CPL=0 and perform one of the two methods noted above. This is an expensive (performance wise) undertaking and is fraught with problems. You would then have to switch back to long mode when finished.
If there is some use case for doing this and you are on a system with multiple cores - You can bring up one of the cores in 32-bit protected mode while the Bootstrap Processor (BSP) runs in long mode.
Method 1: use IRET to enter v8086 mode
This is the easiest solution. If you do an IRET from 32-bit protected mode (in CPL=0) and the EFLAGS.VM register on the stack is set, the CPU will attempt to return to v8086 mode and assumes the stack frame contains the required information to make that transition:
PROTECTED-MODE:
[snip]
EIP ← Pop();
CS ← Pop(); (* 32-bit pop, high-order 16 bits discarded *)
tempEFLAGS ← Pop();
[snip]
RETURN-TO-VIRTUAL-8086-MODE:
(* Interrupted procedure was in virtual-8086 mode: PE = 1, CPL=0, VM = 1 in flag image *)
IF EIP not within CS limit
THEN #GP(0); FI;
EFLAGS ← tempEFLAGS;
ESP ← Pop();
SS ← Pop(); (* Pop 2 words; throw away high-order word *)
ES ← Pop(); (* Pop 2 words; throw away high-order word *)
DS ← Pop(); (* Pop 2 words; throw away high-order word *)
FS ← Pop(); (* Pop 2 words; throw away high-order word *)
GS ← Pop(); (* Pop 2 words; throw away high-order word *)
CPL ← 3;
(* Resume execution in Virtual-8086 mode *)
END;
If you push these items on the stack in reverse order and do the iret you should be able to enter v8086 mode.
V86_STACK_SEG EQU 0x0000 ; v8086 stack SS
V86_STACK_OFS EQU 0x0000 ; v8086 stack SP
V86_CS_SEG EQU 0x0000 ; v8086 code segment CS
EFLAGS_VM_BIT EQU 17 ; EFLAGS VM bit
EFLAGS_BIT1 EQU 1 ; EFLAGS bit 1 (reserved , always 1)
[snip]
xor ebx, ebx ; EBX=0
push ebx ; Real mode GS=0
push ebx ; Real mode FS=0
push ebx ; Real mode DS=0
push ebx ; Real mode ES=0
push V86_STACK_SEG
push V86_STACK_OFS ; v8086 stack SS:SP (grows down from SS:SP)
push dword 1<<EFLAGS_VM_BIT | 1<<EFLAGS_BIT1
; Set VM Bit, IF bit is off, DF=0(forward direction),
; IOPL=0, Reserved bit (bit 1) always 1. Everything
; else 0. These flags will be loaded in the v8086 mode
; during the IRET. We don't want interrupts enabled
; because we have no v86 monitor via protected mode
; GPF handler
push V86_CS_SEG ; Real Mode CS (segment)
push v86_mode_entry ; Entry point (offset)
iret ; Transfer control to v8086 mode and our real mode code
I have set ES=DS=CS=FS=GS=0 and a real mode stack at V86_STACK_SEG:V86_STACK_OFS (define these as you see fit). IP is set to the offset of the v86_mode_entry label. In the code snippet above I only set 2 bits to 1 (bit 1 and VM). Bit 1 is a reserved bit in EFLAGS that is always suppose to be set to 1. All other flags in EFLAGS are 0, thus IOPL=0.
All other registers will contain the same values they had before entering v8086 mode. You may wish to zero them out to avoid leaking information into the v8086 task from 32-bit protected mode (ie: a kernel).
A minimal complete verifiable example of using this code is:
VIDEO_TEXT_ADDR EQU 0xb8000 ; Hard code beginning of text video memory
ATTR_BWHITE_ON_GREEN EQU 0x2f ; Bright white on green attribute
ATTR_BWHITE_ON_MAGENTA EQU 0x5f ; Bright White on magenta attribute
PM_MODE_STACK EQU 0x80000 ; Protected mode stack below EBDA
V86_STACK_SEG EQU 0x0000 ; v8086 stack SS
V86_STACK_OFS EQU 0x0000 ; v8086 stack SP
V86_CS_SEG EQU 0x0000 ; v8086 code segment CS
EFLAGS_VM_BIT EQU 17 ; EFLAGS VM bit
EFLAGS_BIT1 EQU 1 ; EFLAGS bit 1 (reserved, always 1)
EFLAGS_IF_BIT EQU 9 ; EFLAGS IF bit
; Macro to build a GDT descriptor entry
%define MAKE_GDT_DESC(base, limit, access, flags)
(((base & 0x00FFFFFF) << 16) |
((base & 0xFF000000) << 32) |
(limit & 0x0000FFFF) |
((limit & 0x000F0000) << 32) |
((access & 0xFF) << 40) |
((flags & 0x0F) << 52))
bits 16
ORG 0x7c00
; Include a BPB (1.44MB floppy with FAT12) to be more compatible with USB floppy media
; %include "bpb.inc"
boot_start:
xor ax, ax ; DS=SS=ES=0
mov ds, ax
mov ss, ax ; Stack at 0x0000:0x7c00
mov sp, 0x7c00
cld ; Set string instructions to use forward movement
; Fast method of enabling A20 may not work on all x86 BIOSes
; It is good enough for emulators and most modern BIOSes
; See: https://wiki.osdev.org/A20_Line
cli ; Disable interrupts for rest of code as we don't
; want A20 code to be interrupted. In protected mode
; we have no IDT so any interrupt that does occur will
; double fault and reboot.
in al, 0x92
or al, 2
out 0x92, al ; Enable A20 using Fast Method
lgdt [gdtr] ; Load our GDT
mov eax, cr0
or eax, 1
mov cr0, eax ; Set protected mode flag
jmp CODE32_SEL:start32 ; FAR JMP to set CS
; v8086 code entry point
v86_mode_entry:
sub dword [vidmem_ptr], VIDEO_TEXT_ADDR
; Adjust video pointer to be relative to beginning of
; segment 0xb800
mov si, in_v86_msg ; Print in v86 message
mov ah, ATTR_BWHITE_ON_MAGENTA
; Attribute to print with
call print_string_rm_nobios
.endloop:
jmp $ ; Infinite loop since we did code a solution to exit VM
; Function: print_string_rm_nobios
; Display a string to the console on display page 0 in real/v8086 mode
; without using the BIOS. We don't have a proper v8086 monitor so can't
; use BIOS to display.
;
; Very basic. Doesn't update hardware cursor, doesn't handle scrolling,
; LF, CR, TAB.
;
; Inputs: SI = Offset of address to print
; AH = Attribute of string to print
; Clobbers: None
; Returns: None
print_string_rm_nobios:
push di
push si
push ax
push es
mov di, VIDEO_TEXT_ADDR>>4 ; ES=0xb800 (text video mode segment)
mov es, di
mov di, [vidmem_ptr] ; Start from video address stored at vidmem_ptr
jmp .getchar
.outchar:
stosw ; Output character to display
.getchar:
lodsb ; Load next character from string
test al, al ; Is character NUL?
jne .outchar ; If not, go output character
mov [vidmem_ptr], di ; Update global video pointer
pop es
pop ax
pop si
pop di
ret
; 32-bit protected mode entry point
bits 32
start32:
mov ax, DATA32_SEL ; Setup the segment registers with data selector
mov ds, ax
mov es, ax
mov ss, ax
mov esp, PM_MODE_STACK ; Set protected mode stack pointer
mov fs, ax ; Not currently using FS and GS
mov gs, ax
mov ah, ATTR_BWHITE_ON_GREEN; Attribute to print with
mov al, ah ; Attribute to clear last line when scrolling
mov esi, in_pm_msg ; Print message that we are in protected mode
call print_string_pm
xor ebx, ebx ; EBX=0
push ebx ; Real mode GS=0
push ebx ; Real mode FS=0
push ebx ; Real mode DS=0
push ebx ; Real mode ES=0
push V86_STACK_SEG
push V86_STACK_OFS ; v8086 stack SS:SP (grows down from SS:SP)
push dword 1<<EFLAGS_VM_BIT | 1<<EFLAGS_BIT1
; Set VM Bit, IF bit is off, DF=0(forward direction),
; IOPL=0, Reserved bit (bit 1) always 1. Everything
; else 0. These flags will be loaded in the v8086 mode
; during the IRET. We don't want interrupts enabled
; because we have no v86 monitor via protected mode
; GPF handler
push V86_CS_SEG ; Real Mode CS (segment)
push v86_mode_entry ; Entry point (offset)
iret ; Transfer control to v8086 mode and our real mode code
; Function: print_string_pm
; Display a string to the console on display page 0 in protected mode.
; Very basic. Doesn't update hardware cursor, doesn't handle scrolling,
; LF, CR, TAB.
;
; Inputs: ESI = Offset of address to print
; AH = Attribute of string to print
; Clobbers: None
; Returns: None
print_string_pm:
push edi
push esi
push eax
mov edi, [vidmem_ptr] ; Start from video address stored at vidmem_ptr
jmp .getchar
