Not able to open/see assembly instruction and IR details on the generated 'hello.cfg' and 'hello.bc' file with `mcsema-dyninst-disass' as front-end

[ajaymas]

Hello,
I have used mcsema-dyninst-disass as front-end module to generate a CFG (disassembled instruction) file for the 32bit elf binaries using belwo commands.
$ mcsema-dyninst-disass --binary hello --output hello.cfg

I could see the hello.cfg generated file, but i am not able to open and see the CFG or instruction block information on the generated hello.cfg file ( something like IDA pro provided disassembly).
If i try to open, i can see some 0 and 1 information as similar as machine code file opening.

Similarly, I tried to generate a hello.bc or IR file while providing hello.cfg as input using the below command.
$ mcsema-dyninst-disass --binary hello --output hello.cfg

I got below error as well as hello.bc file is generated but still i cant open and see the hello.bc file information. If i try to open, same i can see some 0 and 1 information as similar as machine code file opening in the hello.bc file as well.

Errors:

E20200708 19:08:18.032294 3635 Function.cpp:842] Block 500 has no terminator, and instruction at 500 is not a local no-return function call.
E20200708 19:08:18.034628 3635 Function.cpp:921] Function sub_520_printf is empty!
E20200708 19:08:18.034796 3635 Function.cpp:353] Cannot find target of instruction at 530; the static target 536 is not associated with a lifted subroutine, and it does not have a known call target.
E20200708 19:08:18.034807 3635 Function.cpp:376] Adding missing block 536 in function sub_530_targ530 as a tail call to __remill_error
E20200708 19:08:18.036600 3635 Function.cpp:353] Cannot find target of instruction at 594; the static target 596 is not associated with a lifted subroutine, and it does not have a known call target.
E20200708 19:08:18.036615 3635 Function.cpp:376] Adding missing block 596 in function sub_570_deregister_tm_clones as a tail call to __remill_error
E20200708 19:08:18.036934 3635 Function.cpp:353] Cannot find target of instruction at 5e5; the static target 5e7 is not associated with a lifted subroutine, and it does not have a known call target.
E20200708 19:08:18.036947 3635 Function.cpp:376] Adding missing block 5e7 in function sub_640_frame_dummy as a tail call to __remill_error
E20200708 19:08:18.037375 3635 Function.cpp:353] Cannot find target of instruction at 5e5; the static target 5e7 is not associated with a lifted subroutine, and it does not have a known call target.
E20200708 19:08:18.037389 3635 Function.cpp:376] Adding missing block 5e7 in function sub_5b0_register_tm_clones as a tail call to __remill_error

[pgoodman]

The file a Google Protocol Buffer. The format can be found in mcsema/CFG/CFG.proto.

A .bc file is a bitcode file, which is in a binary format. You will need llvm-dis to view it.

[ajaymas]

Hello @pgoodman
Thanks for your response, I saw the file you have mentioned i.e mcsema/CFG/CFG.proto. If i am not wrong, this is to generate a custom file format on raw CFG file (provided by Ninja or IDA.).
How to see the disassembled instruction or flow graph content of input binary on the generated hello.CFG...?

I have 2 more question:

1. Can we lift LLVM IR from the disassembled assembly instruction (provided by objdump or IDA test mode) instead of CFG format..?

2. What is the advantages to convert raw CFG (provided by IDA pro or Ninja) to a custom file format of CFG generated by google protobuf.

Thanks in advance.

[pgoodman]

1. That is not something that we support. There may be ways of doing it, e.g. with remill-lift or anvill-lift.
2. Having a representation like the CFG file lets us have multiple tools produce these files (right now, only IDA Pro and Dyninst), and support lifting from that file format.

[ajaymas]

@pgoodman I had gone through your first suggested (above) solution using below link.
https://github.com/lifting-bits/remill/blob/master/docs/LIFE_OF_AN_INSTRUCTION.md
I understand remill_lift the LLVM IR from the raw machine code instruction.

However, i still need to understand,

1. How this machine code are disassembled pertaining to Remill..?. Is this machine code were presented inside the CFG file (ida or dyninst provided)...? or it is separate from the CGF file.?

2. Could you mention the usage / execution steps of remill_lift on the raw machine instruction till to generate IR.
   (How i should give raw machine code file of some elf binary as input to remill_lift to generate IR as similar to how i gave hello.CFG as input to generate hello.bc in mcsema_lift)

I committed to work on remil with raw disassembly/ machine instruction code (provided by some objdump disassembler) for elf binary i need your guidance.

Thanks Goodman,

[pgoodman]

McSema's CFG files contains the raw bytes/data of each loaded segment/section from a binary. This data is imported into a NativeModule structure, which extends from an anvill::Program. An anvill::Program has the notion of an address space and memory maps. This happens here: https://github.com/lifting-bits/mcsema/blob/api_improvements/mcsema/CFG/CFG.cpp#L999-L1005

The CFG file also contains a list of functions, their presumed basic blocks, and other details. Most of these are imported, e.g. function entries in the CFG function become NativeFunction in memory in McSema. Eventually, we iterate over all of these and try to lift each of them individually. This happens here: https://github.com/lifting-bits/mcsema/blob/api_improvements/mcsema/BC/Function.cpp#L1728-L1735

The function-specific lifting initially fills a work list with all the basic block heads that were imported from the CFG file (https://github.com/lifting-bits/mcsema/blob/api_improvements/mcsema/BC/Function.cpp#L1507-L1520). These are treated as hints, in the sense that McSema will sometimes go and disover other basic block heads that it needs to lift that weren't originally associated with the function.

Although the work list starts off as "block granularity," really, it's an instruction work list. We start with the address of a block entrypoint, which is popped off the work list. Then lift that instruction if we haven't yet. Lifting that instruction might add new things to our work list, e.g. the next instruction in the block.

Anyway, we try to decode each instruction in the work list, and if we decoded them, then we lift them. This happens here: https://github.com/lifting-bits/mcsema/blob/api_improvements/mcsema/BC/Function.cpp#L1589-L1611 Where do the decoded bytes come from? They were previously "mapped into" the NativeModule (anvill::Program) and so we invoke APIs to read out the bytes, starting at the address of the instruction taken off the work list: https://github.com/lifting-bits/mcsema/blob/api_improvements/mcsema/BC/Function.cpp#L537-L566

So then we have a decoded instruction, which is a remill::Instruction, and we pass it into an instruction lifter. McSema has its own extension to remill's InstructionLifter class. It exists to support things like cross-references, which are something Remill doesn't natively understand. You can find that logic in https://github.com/lifting-bits/mcsema/blob/api_improvements/mcsema/BC/Instruction.cpp

What this all does is fill in llvm::BasicBlocks inside the lifted function with loads/stores to the State structure, as well as calls to semantics functions taken from a loaded semantics module. Optimization of the bitcode inlines all of these functions into the lifted function, and then even more optimizations do things like deferred resolution of cross-reference targets.

[Stephen-lei]

Hi , i still don`t know how to read the .cfg file, can you tell me how to open it? I use vim, hex reader,and I see a lot of 01010100111, thanks!

[pgoodman]

@Stephen-lei it is a protocol-buffer encoded file. Look for CFG_pb2.py somewhere in the codebase, and you can use Python to import the protobuf libraries and parse the file. Another way is to read it with C++. What you could probably do is modify CFG.cpp in mcsema to printed out the DebugString() of the protobuf, if you want to read it as a text file.

[Stephen-lei]

@pgoodman thank you very much!
now I can see what`s in the cfg file!

[pgoodman]

Fantastic! If this a script or tool that you would like to submit for PR? It's worth reading through CFG/CFG.cpp as the CFG file often has a variety of possible errors / things to fix up.

[Stephen-lei]

Fantastic! If this a script or tool that you would like to submit for PR? It's worth reading through CFG/CFG.cpp as the CFG file often has a variety of possible errors / things to fix up.

I agree with your opinion！ There are some bugs in my code, when I finish it I will submit it.