?

Log in

No account? Create an account

[icon] One for the unix geeks - Patti
View:Recent Entries.
View:Archive.
View:Friends.
View:Profile.
View:Website (pattib.org).

Security:
Subject:One for the unix geeks
Time:07:54 pm
This is a head-scratcher.

I have two directories of code, call them Old and New. The difference between Old and New is that I've checked out Old and built the code multiple times. I've checked out New and built the code only once. This code is a library of functions used by a larger application.

If I run using Old, the application fails. If I run using New, it works.

I've compared checksums for every single file in both Old and New, and they're 100% identical. I've diffed the files, and they're identical. I've compared every single file permission, and they too are identical. I've looked for symlinks in the directories, and there are zero. I've even gone so far as to do an ls -lR on each directory, mask out the dates, and diff those, and they're identical.

In desperation, I tarred up Old and New, then untarred them as OldTar and NewTar. Both of those work just fine.

I've done "cp -rP Old OldCopy" and used OldCopy, and that works just fine.

Anybody have any idea what the fuck?


Edit: Sorry, I was unclear. They're libraries of programmatically-generated PHP code, not binaries.

It's a web app. The failure is that one of the php files can't be found, even though it's sitting right there on the disk where it belongs.
comments: Leave a comment Previous Entry Share Next Entry


adbjupe
Link:(Link)
Time:2009-03-02 04:41 am (UTC)
Are the library search paths messing with you? LD_LIBRARY_PATH on Solaris fo instance? Hardwired dynamic link paths in the link step? Do you catually create a library (.a or .so), dynamic or not, from Old and New?
(Reply) (Thread)


whipartist
Link:(Link)
Time:2009-03-02 05:30 am (UTC)
Sorry, I was unclear. It's machine-generated php files, not binaries.
(Reply) (Parent) (Thread)


tigerknight
Link:(Link)
Time:2009-03-02 04:42 am (UTC)
What kind of failure are you getting from the code?
(Reply) (Thread)


whipartist
Link:(Link)
Time:2009-03-02 05:31 am (UTC)
Can't find a (PHP) file that's sitting right there on disk where it belongs, and is 100% perfectly accessible to the app that's looking for it. The error is a little complex, but boils down to, "Dude! I can't find that file!"
(Reply) (Parent) (Thread)


mrtaki
Link:(Link)
Time:2009-03-02 04:47 am (UTC)
Must be something in your environment as Jupe suggested.
If you are using linux, can you run ldd to see which libraries are being referenced?
(Reply) (Thread)


whipartist
Link:(Link)
Time:2009-03-02 05:31 am (UTC)
I was unclear. It's "libraries" of machine-generated PHP files, not binaries.
(Reply) (Parent) (Thread)


andrewhime
Link:(Link)
Time:2009-03-02 05:38 am (UTC)
It's open source!
(Reply) (Thread)

(Deleted comment)

ts4z
Link:(Link)
Time:2009-03-02 08:25 am (UTC)
You have state, probably outside of the filesystem, that's stuck. Perhaps a server has cached the non-existence of some file.

Perhaps you need to alt-reload on your browser or otherwise cripple caching (add a ?fmh=1 to the URL).

Can you rename Old to OldRenamed and try the build that way?

Can you restart a web server process that hasn't beeen restarted?

How do builds/runs work?
(Reply) (Thread)


whipartist
Link:(Link)
Time:2009-03-02 10:46 am (UTC)
The builds are a shell script. They basically delete the machine-generated files, read some stuff from the database, then rebuild the machine-generated files. It's a completely repeatable process-- if you start with the same data, you get exactly the same generated files.

(The stuff pulled from the database is just the structure of the database, not the contents.)

Restarting apache makes the problem go away. My working hypothesis is that apache is somehow caching an inode or something. Perhaps if the inode doesn't exist anymore, the system looks for the file by name, but if it does exist it uses that file... even if the contents have changed. I dunno. That's my handwaving working hypothesis.
(Reply) (Parent) (Thread)


787style
Link:(Link)
Time:2009-03-02 01:55 pm (UTC)
Any hard coded paths (ie looking for /path/to/Old)?
(Reply) (Thread)


whipartist
Link:(Link)
Time:2009-03-02 06:38 pm (UTC)
Nope. I've moved them around by renaming Old to OldX and New to Old, and that still fails.
(Reply) (Parent) (Thread)


yesthattom
Link:(Link)
Time:2009-03-02 02:45 pm (UTC)
Is there a process that has cd'ed into one directory and is still there (even though the directory name has changed?)

Are the permissions on all the parent directories (all the way to root) ok?

Are the 2 directories on the same file system? If they are on different file systems (or servers) is one of them configured differently?

(Reply) (Thread)


whipartist
Link:(Link)
Time:2009-03-02 06:38 pm (UTC)
They're on the same filesystem. The permissions are absolutely 100% identical.

Restarting apache makes it go away.
(Reply) (Parent) (Thread)


yesthattom
Link:(Link)
Time:2009-03-04 12:26 pm (UTC)
Is the process:

1. mkdir dir1
2. start apache
3. apache opens a filedescriptor in dir1 for fast access.
4. mkdir dir2
5. mv dir1 dir1.old
6. mv dir2 dir1
7. apache needs to access files in "dir1" but is smart enough to know it has a file descriptor in dir1, so it fchdir's to the descriptor. The descriptor points at dir1.old because of how inodes work.
8. access via http sees data out of dir1.old... seems like a bug.
9. restart apache. apache closes file descriptor, re-gains descriptor in what is now dir1
10. everything seems to be normal now.

(Reply) (Parent) (Thread)

(Deleted comment)

whipartist
Link:(Link)
Time:2009-03-02 06:36 pm (UTC)
There are no symlinks. I've neurotically checked permissions, as well as every single byte in every single file in the directories, and they're identical.

My guess is that something is caching an inode somewhere, and if the file doesn't exist at that inode it shits itself. Restarting apache makes it go away.
(Reply) (Parent) (Thread)


evwhore
Link:(Link)
Time:2009-03-03 12:54 am (UTC)
Doesn't cp -rP Old OldCopy debunk the idea that it's an inode?

Query: are Old or New actually referenced anywhere in the URL paths that get created, etc.? My guess is some sort of rewrite or redirect that gets applied to New, changing the file system path into something non-existent.

You said in another reply that the error "boils down to." Well, obviously the devil is in the details, and all we can do otherwise is make these broad sweeping guesses, so...
(Reply) (Thread)


whipartist
Link:(Link)
Time:2009-03-03 01:35 am (UTC)
I've tried renaming Old and New in various ways, so I'm confident that it's not anything to do with paths.

The error is:

Fatal Error
Class file for 'TMap' cannot be found.

Given that I later learned that restarting apache makes the problem disappear, I'm still leaning toward the theory that something is being cached when it shouldn't be.


(Reply) (Parent) (Thread)


evwhore
Link:(Link)
Time:2009-03-03 01:53 am (UTC)
That seems plausible.

You mention restarting Apache makes it go away -- how do you get the problem to show up again?
(Reply) (Parent) (Thread)


whipartist
Link:(Link)
Time:2009-03-03 01:56 am (UTC)
Sometimes, seemingly-randomly, when someone rebuilds the code they run into that error.

My guess is that it has to do with whether other files were mucked with on the file system. The build is "rm build_directory; build_into build_directory". It seems likely that if you just do that over and over, everything sort of lands in the same place, but if you do a build, muck about for a while, then do another build, they wind up in different places.

It doesn't explain why tar/untar makes it work though.
(Reply) (Parent) (Thread)

[icon] One for the unix geeks - Patti
View:Recent Entries.
View:Archive.
View:Friends.
View:Profile.
View:Website (pattib.org).