If you would like to trigger the Login or Register popups by a menu item you can do it by add a custom CSS class to the menu item. You can also do it from any custom codes, just add those class to the elements.
In case of Login select "User Manager / Login Form" for menu type at "Link Type / Link CSS Style" write this: log-popup
In case of Registration select "User Manager / Registration Form" for menu type at "Link Type / Link CSS Style" write this: reg-popup
And finally be sure that Improved AJAX Login Register module is enabled on all pages.