Skip to content

/ codeql-cli-binaries

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Some files missing after database creation #14

Open
mvanotti opened this issue Jan 25, 2020 · 11 comments
Open

Some files missing after database creation #14

mvanotti opened this issue Jan 25, 2020 · 11 comments
Assignees
@matt-gretton-dann
Labels
CLI

Comments

@mvanotti
Copy link

@mvanotti mvanotti commented Jan 25, 2020

While trying to create a database for the fuchsia operating system, it seems to leave out some of the files that were built.

Note that the steps to build fuchsia and a codeql database require ~200GB of disk space. I just want to get a better understanding of why it might be skipping some of the files.

Steps to reproduce:

  1. Follow steps on the fuchsia site to download an build fuchsia
  2. Inside the fuchsia directory, run: fx set workstation.x64 --with-base //bundles:kitchen_sink --no-goma
  3. Build the database: codeql database create fuchsia-ql --language=cpp --source-root=${FUCHSIA_DIR} --command="./scripts/fx clean-build"
  4. List the files with a script similar to this one
  5. List all the fuchsia cc, ignoring the third_party files: find ${FUCHSIA_DIR} -iname "*.cc" | grep -v "third_party"
  6. Compare the files in the database with the cc files. Some of the files are missing.

This is the list that I have in my system. In particular, files like /zircon/tools/zbi/zbi.cc should always be built.

To get a list of all the compiled files in fuchsia, run: fx compdb and then look at out/default/compile_commands.json and out/default.zircon/compile_commands.json
@alexet
Copy link

@alexet alexet commented Jan 27, 2020

We crash while extracting that file so it doesn't end up in the database. We can reproduce it and we are working on a fix.
@matt-gretton-dann
Copy link

@matt-gretton-dann matt-gretton-dann commented Jan 28, 2020

Can you please give the output of codeql --version? There are several different issues going on here and I want to check that I am seeing the same issues you are.
@mvanotti
Copy link
Author

@mvanotti mvanotti commented Jan 29, 2020

Hi Matt,

$ ./codeql/codeql --version
CodeQL command-line toolchain.
Version: 2.0.1.
Copyright (C) 2019 GitHub, Inc.

I'm using the latest released version downloaded from this github repo
@matt-gretton-dann
Copy link

@matt-gretton-dann matt-gretton-dann commented Jan 31, 2020

Thank you for that info. I have identified several different issues which in our code base which are being triggered by the fuchsia build. I am working on fixes - but unfortunately they are unlikely to land in the next release of CodeQL. I will keep you updated on how the work is going through this ticket.
@mvanotti
Copy link
Author

@mvanotti mvanotti commented Feb 2, 2020

Cool! Let me know if there's anything I can do to help.
@matt-gretton-dann
Copy link

@matt-gretton-dann matt-gretton-dann commented Feb 11, 2020

We have fixes internally now for a couple of the main issues (lld not recognised as a linker, and the vast majority of extraction failures). These should make it into the next release of the CodeQL CLI Binaries (v2.0.3). I do not have a date for that release yet.
@hmakholm
Copy link
Contributor

@hmakholm hmakholm commented Feb 11, 2020

(@matt-gretton-dann actually 2.0.3 will probably be a point release after 2.0.2 to fix just a single brown-paper-bag bug, and probably out tomorrow. The one that contains your fixes this will then be 2.0.4).
@mvanotti
Copy link
Author

@mvanotti mvanotti commented Apr 16, 2020

Do you know if the change made it a release? We are at 2.1.0
@mvanotti
Copy link
Author

@mvanotti mvanotti commented Apr 16, 2020

/zircon/tools/zbi/zbi.cc is now being included in my database. Will continue testing to see if there's something else missing.
@mvanotti
Copy link
Author

@mvanotti mvanotti commented Apr 16, 2020

I ran the following query to list all the files:

import cpp

from File f
select f.getAbsolutePath(), ""

I see 8046 different files.

And also gathered all the files from compile_commands.json used during compilation in fuchsia (both in out/default and out/default.zircon). Doing a sort | uniq, I get 9320 different cc files.

This means that there are ~1.3k files that are missing. This gist contains the list of files in the db and in compile_commands.json.

Some examples of missing files (by eye) are some of the autogenerated fidl files, some unit tests, and some random files that I have no idea why are missing.
@adityasharad adityasharad added the CLI label Jul 31, 2020
@mvanotti
Copy link
Author

@mvanotti mvanotti commented Aug 17, 2020

Hi! I've re-ran this test with CodeQL CLI version 2.2.4, and found similar results.

By looking only at c and cc files, I see that there are 1140 missing files from the database, albeit ~700 of them are test files, but they do appear in compile_commands.json.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@matt-gretton-dann matt-gretton-dann

Labels
CLI
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
@mvanotti @alexet @adityasharad @hmakholm @matt-gretton-dann
You can’t perform that action at this time.